{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、数据集切分\n",
    "### 交叉验证\n",
    "10W训练集分成8W和2W,8W用于第一层模型交叉验证，2W用于第二层模型交叉验证\n",
    "\n",
    "## 二、算法流程\n",
    "### 预处理\n",
    "1. 用LR填充空缺目标值，合并train test数据集\n",
    "2. 训练dbow和dm ２种doc2vec特征\n",
    "\n",
    "### 训练第一层模型\n",
    "1. 训练tfidf-lr模型\n",
    "2. 训练dbow-nn模型\n",
    "3. 训练dm-nn模型\n",
    "\n",
    "### 训练第二层模型\n",
    "1. 训练 xgb-ens模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache C:\\Users\\lhc\\AppData\\Local\\Temp\\jieba.cache\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(100000, 5)\n",
      "(100000, 2)\n",
      " 4    37107\n",
      " 3    28148\n",
      " 2    18858\n",
      "-1     9280\n",
      " 5     5693\n",
      " 1      560\n",
      " 0      354\n",
      "Name: Education, dtype: int64\n",
      " 0    38996\n",
      " 1    26744\n",
      " 2    18529\n",
      " 3    10654\n",
      " 4     2922\n",
      "-1     1666\n",
      " 5      489\n",
      "Name: age, dtype: int64\n",
      " 0    56976\n",
      " 1    40869\n",
      "-1     2155\n",
      "Name: gender, dtype: int64\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading model cost 0.735 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000 20000 30000 40000 50000 60000 丰田中东和中规\t江西五十铃\t奇瑞\t江淮\t帕杰罗劲畅\t江玲好还是五十铃好\t打大树和打沙袋的区别\t一个月赚多少钱能开起霸道\t商品房合同上面写享受房交会价钱\t天籁和雅阁哪个好\t驭胜乞丐版\t微信摇一摇骗子\t2016款途观\t江铃驭胜怎么样\t创步沙袋\t租车有什么说道吗\t底盘很高的越野车\t福特猛禽\t鞍山哪检三轮车\t炖鱼为什么会碎\t福特撼路者\t星月圆珠子都是水磨吗\t斯里兰卡好玩吗 沙滩\t莲子心泡水喝的功效\t全款买房延期怎么算\t十万元左右的suv车\t酷路泽4000中东版价格\t江西五十铃mu x视频\t驭胜s3502016款柴油\t电脑无限封逆战号\t丰田\t呐志捷大7\t鞍山租车凯美瑞\t大显suv\t驭胜有四驱的吗\t架子鼓女神雅研\t十万左右的二手suv\t广汽传琪\t驭胜新款好还是老款好\t驭胜\t中华十一绝牛人都是谁\t迈腾\tmg\t驭胜好吗\t奇骏2.5都是一样的发动机吗\t鞍山铁西教散打的\t鞍山靓号网\t大石桥车能在鞍山检车吗\t江西五十铃mx\t本田crv\t鞍山江西五十铃\t立式沙袋一般多高\t鱼怎么炖好吃\t很大的吉普不是很贵\t中牧水龙头\t鞍山租车\t一汽丰田和广汽丰田哪个好\t江淮大好运4.2米报价和车图\t五十铃\t鞍山五十铃4s店地址\t北京40视频在线观看\t索兰托\t中华十一绝老大是谁\t北京b40\t二手霸道2700\t本田几代好\t雅阁方向机有问题吗\t江玲\t沙袋太矮怎么打\t2016suv销量排行榜\t煤油打火机调火石\t大力金刚腿怎么练\t江西五十铃mu-x好不好\t呐志杰u7\t二手霸道值得入手吗\ts350驭胜\t奇骏没防撞梁\t中华十一绝牛人排行\t本田雅阁几代最好\t瑞麒\t2700霸道废油吗\t18万买什么suv车好\t中华十一绝都有谁\t乞丐版途观好不好\t铁砂掌怎么练\tjeep\t本田crv好不好\t莲子鱼的做法大全\t中华十一绝人物介绍\t2,\t十万买什么轿车最经济\t鞍山铁西教拳击的\t连子鱼怎么做好吃\t奇骏大事故到底抗撞吗\t最保值的suv\t凯美瑞好还是雅阁好\t斯巴鲁\t驭胜s350缺点\t驭胜和江西五十铃\t手动新途观\t苹果四声音太小怎么办\t斯里兰卡是哪个国家\t暂停服务是什么意思\t大众途观1.8手动\t奥迪a4\t雅阁九代有问题吗\t五菱荣光换怠速马达多少钱\t可以越野中型suv车型\t五菱荣光换胎速马达多少钱\t鞍山教自由搏击的\t呐志捷\t江西五十铃mu-x\t学卖二手车好学吗\t中牧和伟星\t越野车\t硬派越野车\t新途观什么时候上市\t十多万能越野的车\t大众\t起亚\t微信转账后发现被骗能否找回\t一米五沙袋适合多高个子打\t鞍山江玲4s店\t13610984444\t北京40l\t2.5奇骏通病\t马又念什么\t江玲驭胜\t北京40l怎么样\tsuv销量排行榜\t鞍山有江西五十铃吗\t三菱越野10万元的车\t老款汉兰达好吗\t青云志小七是谁\t鞍山哪能检三轮摩托\t博美被妈妈咬了怎么好的快\t奇骏钢板很薄\n",
      "\n",
      "====================\n",
      "['丰田', '中东', '和', '中规', '丰田_*_中东', '中东_*_和', '和_*_中规', '江西', '五十铃', '江西_*_五十铃', '奇瑞', '江淮', '帕杰罗', '劲畅', '帕杰罗_*_劲畅', '江玲', '好', '还是', '五十铃', '好', '江玲_*_好', '好_*_还是', '还是_*_五十铃', '五十铃_*_好', '打', '大树', '和', '打', '沙袋', '的', '区别', '打_*_大树', '大树_*_和', '和_*_打', '打_*_沙袋', '沙袋_*_的', '的_*_区别', '一个月', '赚', '多少', '钱能', '开起', '霸道', '一个月_*_赚', '赚_*_多少', '多少_*_钱能', '钱能_*_开起', '开起_*_霸道', '商品房', '合同', '上面', '写', '享受', '房交会', '价钱', '商品房_*_合同', '合同_*_上面', '上面_*_写', '写_*_享受', '享受_*_房交会', '房交会_*_价钱', '天籁', '和', '雅阁', '哪个', '好', '天籁_*_和', '和_*_雅阁', '雅阁_*_哪个', '哪个_*_好', '驭', '胜', '乞丐', '版', '驭_*_胜', '胜_*_乞丐', '乞丐_*_版', '微信', '摇一摇', '骗子', '微信_*_摇一摇', '摇一摇_*_骗子', '2016', '款途观', '2016_*_款途观', '江铃驭', '胜', '怎么样', '江铃驭_*_胜', '胜_*_怎么样', '创步', '沙袋', '创步_*_沙袋', '租车', '有', '什么', '说道', '吗', '租车_*_有', '有_*_什么', '什么_*_说道', '说道_*_吗', '底盘', '很', '高', '的', '越野车', '底盘_*_很', '很_*_高', '高_*_的', '的_*_越野车', '福特', '猛禽', '福特_*_猛禽', '鞍山', '哪检', '三轮车', '鞍山_*_哪检', '哪检_*_三轮车', '炖鱼', '为什么', '会碎', '炖鱼_*_为什么', '为什么_*_会碎', '福特', '撼路者', '福特_*_撼路者', '星月', '圆', '珠子', '都', '是', '水磨', '吗', '星月_*_圆', '圆_*_珠子', '珠子_*_都', '都_*_是', '是_*_水磨', '水磨_*_吗', '斯里兰卡', '好玩', '吗', ' ', '沙滩', '斯里兰卡_*_好玩', '好玩_*_吗', '吗_*_ ', ' _*_沙滩', '莲子心', '泡水', '喝', '的', '功效', '莲子心_*_泡水', '泡水_*_喝', '喝_*_的', '的_*_功效', '全款', '买房', '延期', '怎么', '算', '全款_*_买房', '买房_*_延期', '延期_*_怎么', '怎么_*_算', '十万元', '左右', '的', 'suv', '车', '十万元_*_左右', '左右_*_的', '的_*_suv', 'suv_*_车', '酷路泽', '4000', '中东', '版', '价格', '酷路泽_*_4000', '4000_*_中东', '中东_*_版', '版_*_价格', '江西', '五十铃', 'mu', ' ', 'x', '视频', '江西_*_五十铃', '五十铃_*_mu', 'mu_*_ ', ' _*_x', 'x_*_视频', '驭', '胜', 's3502016', '款', '柴油', '驭_*_胜', '胜_*_s3502016', 's3502016_*_款', '款_*_柴油', '电脑', '无限', '封逆战', '号', '电脑_*_无限', '无限_*_封逆战', '封逆战_*_号', '丰田', '呐志捷', '大', '7', '呐志捷_*_大', '大_*_7', '鞍山', '租车', '凯美瑞', '鞍山_*_租车', '租车_*_凯美瑞', '大显', 'suv', '大显_*_suv', '驭', '胜有', '四驱', '的', '吗', '驭_*_胜有', '胜有_*_四驱', '四驱_*_的', '的_*_吗', '架子鼓', '女神', '雅研', '架子鼓_*_女神', '女神_*_雅研', '十万左右', '的', '二手', 'suv', '十万左右_*_的', '的_*_二手', '二手_*_suv', '广汽传琪', '驭', '胜', '新款', '好', '还是', '老款', '好', '驭_*_胜', '胜_*_新款', '新款_*_好', '好_*_还是', '还是_*_老款', '老款_*_好', '驭', '胜', '驭_*_胜', '中华', '十一', '绝牛人', '都', '是', '谁', '中华_*_十一', '十一_*_绝牛人', '绝牛人_*_都', '都_*_是', '是_*_谁', '迈腾', 'mg', '驭', '胜', '好', '吗', '驭_*_胜', '胜_*_好', '好_*_吗', '奇骏', '2.5', '都', '是', '一样', '的', '发动机', '吗', '奇骏_*_2.5', '2.5_*_都', '都_*_是', '是_*_一样', '一样_*_的', '的_*_发动机', '发动机_*_吗', '鞍山', '铁西', '教', '散打', '的', '鞍山_*_铁西', '铁西_*_教', '教_*_散打', '散打_*_的', '鞍山', '靓号', '网', '鞍山_*_靓号', '靓号_*_网', '大石桥', '车能', '在', '鞍山', '检车', '吗', '大石桥_*_车能', '车能_*_在', '在_*_鞍山', '鞍山_*_检车', '检车_*_吗', '江西', '五十铃', 'mx', '江西_*_五十铃', '五十铃_*_mx', '本田', 'crv', '本田_*_crv', '鞍山', '江西', '五十铃', '鞍山_*_江西', '江西_*_五十铃', '立式', '沙袋', '一般', '多高', '立式_*_沙袋', '沙袋_*_一般', '一般_*_多高', '鱼', '怎么', '炖', '好吃', '鱼_*_怎么', '怎么_*_炖', '炖_*_好吃', '很大', '的', '吉普', '不是', '很', '贵', '很大_*_的', '的_*_吉普', '吉普_*_不是', '不是_*_很', '很_*_贵', '中牧', '水龙头', '中牧_*_水龙头', '鞍山', '租车', '鞍山_*_租车', '一汽', '丰田', '和', '广汽', '丰田', '哪个', '好', '一汽_*_丰田', '丰田_*_和', '和_*_广汽', '广汽_*_丰田', '丰田_*_哪个', '哪个_*_好', '江淮', '大', '好运', '4.2', '米', '报价', '和', '车图', '江淮_*_大', '大_*_好运', '好运_*_4.2', '4.2_*_米', '米_*_报价', '报价_*_和', '和_*_车图', '五十铃', '鞍山', '五十铃', '4s店', '地址', '鞍山_*_五十铃', '五十铃_*_4s店', '4s店_*_地址', '北京', '40', '视频', '在线', '观看', '北京_*_40', '40_*_视频', '视频_*_在线', '在线_*_观看', '索兰托', '中华', '十一', '绝', '老大', '是', '谁', '中华_*_十一', '十一_*_绝', '绝_*_老大', '老大_*_是', '是_*_谁', '北京', 'b40', '北京_*_b40', '二手', '霸道', '2700', '二手_*_霸道', '霸道_*_2700', '本田', '几代', '好', '本田_*_几代', '几代_*_好', '雅阁', '方向机', '有', '问题', '吗', '雅阁_*_方向机', '方向机_*_有', '有_*_问题', '问题_*_吗', '江玲', '沙袋', '太矮', '怎么', '打', '沙袋_*_太矮', '太矮_*_怎么', '怎么_*_打', '2016suv', '销量', '排行榜', '2016suv_*_销量', '销量_*_排行榜', '煤油', '打火机', '调', '火石', '煤油_*_打火机', '打火机_*_调', '调_*_火石', '大力', '金刚', '腿', '怎么', '练', '大力_*_金刚', '金刚_*_腿', '腿_*_怎么', '怎么_*_练', '江西', '五十铃', 'mu', '-', 'x', '好不好', '江西_*_五十铃', '五十铃_*_mu', 'mu_*_-', '-_*_x', 'x_*_好不好', '呐志杰', 'u7', '呐志杰_*_u7', '二手', '霸道', '值得', '入手', '吗', '二手_*_霸道', '霸道_*_值得', '值得_*_入手', '入手_*_吗', 's350', '驭', '胜', 's350_*_驭', '驭_*_胜', '奇骏', '没', '防撞', '梁', '奇骏_*_没', '没_*_防撞', '防撞_*_梁', '中华', '十一', '绝牛人', '排行', '中华_*_十一', '十一_*_绝牛人', '绝牛人_*_排行', '本田雅阁', '几代', '最好', '本田雅阁_*_几代', '几代_*_最好', '瑞麒', '2700', '霸道', '废油', '吗', '2700_*_霸道', '霸道_*_废油', '废油_*_吗', '18', '万买', '什么', 'suv', '车好', '18_*_万买', '万买_*_什么', '什么_*_suv', 'suv_*_车好', '中华', '十一', '绝', '都', '有', '谁', '中华_*_十一', '十一_*_绝', '绝_*_都', '都_*_有', '有_*_谁', '乞丐', '版途观', '好不好', '乞丐_*_版途观', '版途观_*_好不好', '铁砂掌', '怎么', '练', '铁砂掌_*_怎么', '怎么_*_练', 'jeep', '本田', 'crv', '好不好', '本田_*_crv', 'crv_*_好不好', '莲子', '鱼', '的', '做法', '大全', '莲子_*_鱼', '鱼_*_的', '的_*_做法', '做法_*_大全', '中华', '十一', '绝', '人物', '介绍', '中华_*_十一', '十一_*_绝', '绝_*_人物', '人物_*_介绍', '2', ',', '2_*_,', '十万', '买', '什么', '轿车', '最', '经济', '十万_*_买', '买_*_什么', '什么_*_轿车', '轿车_*_最', '最_*_经济', '鞍山', '铁西', '教', '拳击', '的', '鞍山_*_铁西', '铁西_*_教', '教_*_拳击', '拳击_*_的', '连子', '鱼', '怎么', '做', '好吃', '连子_*_鱼', '鱼_*_怎么', '怎么_*_做', '做_*_好吃', '奇骏', '大', '事故', '到底', '抗撞', '吗', '奇骏_*_大', '大_*_事故', '事故_*_到底', '到底_*_抗撞', '抗撞_*_吗', '最', '保值', '的', 'suv', '最_*_保值', '保值_*_的', '的_*_suv', '凯美瑞', '好', '还是', '雅阁', '好', '凯美瑞_*_好', '好_*_还是', '还是_*_雅阁', '雅阁_*_好', '斯巴鲁', '驭', '胜', 's350', '缺点', '驭_*_胜', '胜_*_s350', 's350_*_缺点', '驭', '胜', '和', '江西', '五十铃', '驭_*_胜', '胜_*_和', '和_*_江西', '江西_*_五十铃', '手动', '新途观', '手动_*_新途观', '苹果', '四', '声音', '太小', '怎么办', '苹果_*_四', '四_*_声音', '声音_*_太小', '太小_*_怎么办', '斯里兰卡', '是', '哪个', '国家', '斯里兰卡_*_是', '是_*_哪个', '哪个_*_国家', '暂停', '服务', '是', '什么', '意思', '暂停_*_服务', '服务_*_是', '是_*_什么', '什么_*_意思', '大众', '途观', '1.8', '手动', '大众_*_途观', '途观_*_1.8', '1.8_*_手动', '奥迪', 'a4', '奥迪_*_a4', '雅阁', '九代', '有', '问题', '吗', '雅阁_*_九代', '九代_*_有', '有_*_问题', '问题_*_吗', '五菱', '荣光', '换', '怠速', '马达', '多少', '钱', '五菱_*_荣光', '荣光_*_换', '换_*_怠速', '怠速_*_马达', '马达_*_多少', '多少_*_钱', '可以', '越野', '中型', 'suv', '车型', '可以_*_越野', '越野_*_中型', '中型_*_suv', 'suv_*_车型', '五菱', '荣光', '换胎', '速', '马达', '多少', '钱', '五菱_*_荣光', '荣光_*_换胎', '换胎_*_速', '速_*_马达', '马达_*_多少', '多少_*_钱', '鞍山', '教', '自由', '搏击', '的', '鞍山_*_教', '教_*_自由', '自由_*_搏击', '搏击_*_的', '呐志捷', '江西', '五十铃', 'mu', '-', 'x', '江西_*_五十铃', '五十铃_*_mu', 'mu_*_-', '-_*_x', '学卖', '二手车', '好学', '吗', '学卖_*_二手车', '二手车_*_好学', '好学_*_吗', '中牧和伟星', '越野车', '硬派', '越野车', '硬派_*_越野车', '新途观', '什么', '时候', '上市', '新途观_*_什么', '什么_*_时候', '时候_*_上市', '十多万', '能', '越野', '的', '车', '十多万_*_能', '能_*_越野', '越野_*_的', '的_*_车', '大众', '起亚', '微信', '转账', '后', '发现', '被', '骗', '能否', '找回', '微信_*_转账', '转账_*_后', '后_*_发现', '发现_*_被', '被_*_骗', '骗_*_能否', '能否_*_找回', '一米', '五', '沙袋', '适合', '多', '高个子', '打', '一米_*_五', '五_*_沙袋', '沙袋_*_适合', '适合_*_多', '多_*_高个子', '高个子_*_打', '鞍山', '江玲', '4s店', '鞍山_*_江玲', '江玲_*_4s店', '13610984444', '北京', '40l', '北京_*_40l', '2.5', '奇骏', '通病', '2.5_*_奇骏', '奇骏_*_通病', '马', '又', '念', '什么', '马_*_又', '又_*_念', '念_*_什么', '江玲驭', '胜', '江玲驭_*_胜', '北京', '40l', '怎么样', '北京_*_40l', '40l_*_怎么样', 'suv', '销量', '排行榜', 'suv_*_销量', '销量_*_排行榜', '鞍山', '有', '江西', '五十铃', '吗', '鞍山_*_有', '有_*_江西', '江西_*_五十铃', '五十铃_*_吗', '三菱', '越野', '10', '万元', '的', '车', '三菱_*_越野', '越野_*_10', '10_*_万元', '万元_*_的', '的_*_车', '老款', '汉兰达', '好', '吗', '老款_*_汉兰达', '汉兰达_*_好', '好_*_吗', '青云', '志小七是', '谁', '青云_*_志小七是', '志小七是_*_谁', '鞍山', '哪能', '检', '三轮', '摩托', '鞍山_*_哪能', '哪能_*_检', '检_*_三轮', '三轮_*_摩托', '博美', '被', '妈妈', '咬', '了', '怎么', '好', '的', '快', '博美_*_被', '被_*_妈妈', '妈妈_*_咬', '咬_*_了', '了_*_怎么', '怎么_*_好', '好_*_的', '的_*_快', '奇骏', '钢板', '很', '薄', '\\n', '奇骏_*_钢板', '钢板_*_很', '很_*_薄', '薄_*_\\n']\n",
      "70000 80000 90000 100000 1699505\n"
     ]
    }
   ],
   "source": [
    "'''1.concat train data and test data \n",
    "   2.use lr to fill null label'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import jieba\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.cross_validation import StratifiedShuffleSplit,StratifiedKFold,cross_val_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.naive_bayes import MultinomialNB,BernoulliNB,GaussianNB\n",
    "import pickle\n",
    "import Cfg\n",
    "\n",
    "#----------------------load data--------------------------------\n",
    "df_tr = []\n",
    "for i,line in enumerate(open(cfg.data_path + 'user_tag_query.10W.TRAIN',encoding='GB18030')):\n",
    "    segs = line.split('\\t')\n",
    "    row = {}\n",
    "    row['Id'] = segs[0]\n",
    "    row['age'] = int(segs[1])\n",
    "    row['gender'] = int(segs[2])\n",
    "    row['Education'] = int(segs[3])\n",
    "    row['query'] = '\\t'.join(segs[4:])\n",
    "    df_tr.append(row)\n",
    "df_tr = pd.DataFrame(df_tr)\n",
    "\n",
    "df_te=[]\n",
    "for i,line in enumerate(open(cfg.data_path + 'user_tag_query.10W.TEST',encoding='GB18030')):\n",
    "    segs = line.split('\\t')\n",
    "    row = {}\n",
    "    row['Id'] = segs[0]\n",
    "    row['query'] = '\\t'.join(segs[1:])\n",
    "    df_te.append(row)\n",
    "df_te = pd.DataFrame(df_te)\n",
    "\n",
    "print(df_tr.shape)\n",
    "print(df_te.shape)\n",
    "\n",
    "df_all = df_tr\n",
    "# df_all = pd.concat([df_tr,df_te]).fillna(1)\n",
    "# df_all.index = range(len(df_all))\n",
    "\n",
    "for lb in ['Education','age','gender']:\n",
    "    df_all[lb] = df_all[lb] - 1\n",
    "    print(df_all.iloc[:100000][lb].value_counts())\n",
    "    \n",
    "class Tokenizer():\n",
    "    def __init__(self):\n",
    "        self.n = 0\n",
    "    def __call__(self,line):\n",
    "        tokens = []\n",
    "        for query in line.split('\\t'):\n",
    "            words = [word for word in jieba.cut(query)]\n",
    "            for gram in [1,2]:\n",
    "                for i in range(len(words) - gram + 1):\n",
    "                    tokens += [\"_*_\".join(words[i:i+gram])]\n",
    "        if np.random.rand() < 0.00001:\n",
    "            print(line)\n",
    "            print('='*20)\n",
    "            print(tokens)\n",
    "        self.n += 1\n",
    "        if self.n%10000==0:\n",
    "            print(self.n,end=' ')\n",
    "        return tokens    \n",
    "\n",
    "tfv = TfidfVectorizer(tokenizer=Tokenizer(),min_df=3,max_df=0.95,sublinear_tf=True)\n",
    "X_sp = tfv.fit_transform(df_all['query'])\n",
    "# pickle.dump(X_sp,open(root + 'tfidf_10W.pkl','wb'))\n",
    "print(len(tfv.vocabulary_))\n",
    "X_all = X_sp\n",
    "\n",
    "#-----------------------------fill nan-------------------------------------\n",
    "'''填充空值'''\n",
    "for lb,idx in [('Education',0),('age',2),('gender',3)]:\n",
    "    tr = np.where(df_all[lb]!=-1)[0]\n",
    "    va = np.where(df_all[lb]==-1)[0]\n",
    "lb = 'Education'\n",
    "idx = 0\n",
    "tr = np.where(df_all[lb]!=-1)[0]\n",
    "va = np.where(df_all[lb]==-1)[0]\n",
    "df_all.iloc[va,idx] = LogisticRegression(C=1).fit(X_all[tr],df_all.iloc[tr,idx]).predict(X_all[va])\n",
    "\n",
    "lb = 'age'\n",
    "idx = 2\n",
    "tr = np.where(df_all[lb]!=-1)[0]\n",
    "va = np.where(df_all[lb]==-1)[0]\n",
    "df_all.iloc[va,idx] = LogisticRegression(C=2).fit(X_all[tr],df_all.iloc[tr,idx]).predict(X_all[va])\n",
    "\n",
    "lb = 'gender'\n",
    "idx = 3\n",
    "tr = np.where(df_all[lb]!=-1)[0]\n",
    "va = np.where(df_all[lb]==-1)[0]\n",
    "df_all.iloc[va,idx] = LogisticRegression(C=2).fit(X_all[tr],df_all.iloc[tr,idx]).predict(X_all[va])\n",
    "\n",
    "df_all = pd.concat([df_all,df_te]).fillna(0)\n",
    "df_all.to_csv(cfg.data_path + 'all_v2.csv',index=None,encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:840: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n",
      "C:\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:1015: UserWarning: Pattern library is not installed, lemmatization won't be available.\n",
      "  warnings.warn(\"Pattern library is not installed, lemmatization won't be available.\")\n",
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache C:\\Users\\lhc\\AppData\\Local\\Temp\\jieba.cache\n",
      "Loading model cost 0.700 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-01-12 15:36:29.419356 0\n",
      "2017-01-12 15:37:24.815547 10000\n",
      "2017-01-12 15:38:20.066305 20000\n",
      "2017-01-12 15:39:12.412226 30000\n",
      "2017-01-12 15:40:08.696306 40000\n",
      "2017-01-12 15:41:04.784937 50000\n",
      "2017-01-12 15:42:02.058357 60000\n",
      "2017-01-12 15:42:56.711774 70000\n",
      "2017-01-12 15:43:49.831915 80000\n",
      "2017-01-12 15:44:40.984926 90000\n",
      "2017-01-12 15:46:00.153113 pass: 1\n",
      "dbow Education [ 0.62836858  0.62716864  0.62511874  0.62743137  0.63236324] 0.628090115019\n",
      "dbow age [ 0.58209179  0.57754225  0.58277086  0.58365837  0.58643797] 0.582500245917\n",
      "dbow gender [ 0.8343   0.8301   0.8357   0.83895  0.8347 ] 0.83475\n",
      "2017-01-12 16:01:27.135225 pass: 2\n",
      "dbow Education [ 0.63476826  0.6359682   0.6339683   0.63318166  0.63336334] 0.634249952036\n",
      "dbow age [ 0.59129087  0.58759124  0.58482076  0.58730873  0.59193879] 0.588590078489\n",
      "dbow gender [ 0.8367   0.8344   0.83825  0.8417   0.83895] 0.838\n",
      "2017-01-12 16:11:33.578202 save done\n",
      "2017-01-12 16:12:00.556195 pass: 0\n",
      "dm Education [ 0.59062047  0.59892005  0.59537023  0.59052953  0.59570957] 0.594229970379\n",
      "dm age [ 0.5479952   0.5550445   0.55132243  0.5420042   0.55793369] 0.550860004076\n",
      "dm gender [ 0.8028   0.80625  0.80965  0.80745  0.8069 ] 0.80661\n",
      "2017-01-12 16:22:59.573850 pass: 1\n",
      "dm Education [ 0.61156942  0.61451927  0.61531923  0.61308065  0.61531153] 0.613960022958\n",
      "dm age [ 0.56869313  0.56549345  0.57077146  0.56440644  0.57318598] 0.568510092262\n",
      "dm gender [ 0.81915  0.8186   0.82165  0.8236   0.82565] 0.82173\n",
      "2017-01-12 16:32:53.250185 pass: 2\n",
      "dm Education [ 0.62226889  0.62226889  0.62291885  0.62503125  0.62871287] 0.624240150004\n",
      "dm age [ 0.58074193  0.57579242  0.580071    0.57350735  0.58003701] 0.57802993986\n",
      "dm gender [ 0.82655  0.8267   0.83105  0.83255  0.83075] 0.82952\n",
      "2017-01-12 16:42:34.117304 pass: 3\n",
      "dm Education [ 0.62831858  0.62611869  0.62851857  0.6260313   0.6309631 ] 0.627990050016\n",
      "dm age [ 0.58449155  0.58269173  0.58367082  0.57870787  0.58228734] 0.582369862404\n",
      "dm gender [ 0.83135  0.83055  0.8346   0.83665  0.8335 ] 0.83333\n",
      "2017-01-12 16:52:51.099887 pass: 4\n",
      "dm Education [ 0.63021849  0.62891855  0.62726864  0.62983149  0.63286329] 0.629820091524\n",
      "dm age [ 0.58484152  0.58339166  0.58487076  0.57705771  0.58513777] 0.583059881916\n",
      "dm gender [ 0.8333   0.832    0.83565  0.83565  0.8346 ] 0.83424\n",
      "2017-01-12 17:02:15.369150 pass: 5\n",
      "dm Education [ 0.63061847  0.62871856  0.62741863  0.62833142  0.63371337] 0.629760090025\n",
      "dm age [ 0.58619138  0.58509149  0.58472076  0.57950795  0.58648797] 0.584399911933\n",
      "dm gender [ 0.8349   0.83085  0.8363   0.8365   0.83385] 0.83448\n",
      "2017-01-12 17:11:12.076179 pass: 6\n",
      "dm Education [ 0.63171841  0.63131843  0.63016849  0.63178159  0.6380138 ] 0.632600146039\n",
      "dm age [ 0.58514149  0.58779122  0.58662067  0.58040804  0.58613792] 0.585219867438\n",
      "dm gender [ 0.83285  0.8298   0.8361   0.8357   0.8326 ] 0.83341\n",
      "2017-01-12 17:20:03.760705 pass: 7\n",
      "dm Education [ 0.63326834  0.63121844  0.63016849  0.62903145  0.63541354] 0.631820052033\n",
      "dm age [ 0.58674133  0.58594141  0.58532073  0.58220822  0.58633795] 0.585309927441\n",
      "dm gender [ 0.8332   0.82955  0.8365   0.8362   0.83365] 0.83382\n",
      "2017-01-12 17:28:49.958025 pass: 8\n",
      "dm Education [ 0.63021849  0.62971851  0.62756862  0.62973149  0.63641364] 0.630730150531\n",
      "dm age [ 0.5850415   0.5849915   0.58572071  0.58345835  0.5853378 ] 0.584909971434\n",
      "dm gender [ 0.8351   0.82855  0.8354   0.83595  0.83415] 0.83383\n",
      "2017-01-12 17:37:28.490699 pass: 9\n",
      "dm Education [ 0.63101845  0.63111844  0.62926854  0.63188159  0.63551355] 0.631760115033\n",
      "dm age [ 0.58719128  0.58684132  0.58587071  0.5819582   0.58588788] 0.585549876441\n",
      "dm gender [ 0.83205  0.8288   0.8341   0.83525  0.83415] 0.83287\n",
      "2017-01-12 17:46:16.879076 save done\n"
     ]
    }
   ],
   "source": [
    "'''train dbow/dm for education/age/gender'''\n",
    "\n",
    "import pandas as pd\n",
    "import jieba\n",
    "from datetime import datetime\n",
    "from collections import namedtuple\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "import subprocess\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "import codecs\n",
    "import cfg\n",
    "import numpy as np\n",
    "\n",
    "df_all = pd.read_csv(cfg.data_path + 'all_v2.csv',encoding='utf8')\n",
    "#-------------------add row number to query----------------------\n",
    "doc_f = codecs.open('alldata-id.txt','w',encoding='utf8')\n",
    "for i,queries in enumerate(df_all.iloc[:100000]['query']):\n",
    "    words = []\n",
    "    for query in queries.split('\\t'):\n",
    "        words.extend(list(jieba.cut(query)))\n",
    "    tags = [i]\n",
    "    if i % 10000 == 0:\n",
    "        print(datetime.now(),i)\n",
    "    doc_f.write('_*{} {}'.format(i,' '.join(words)))\n",
    "doc_f.close()\n",
    "\n",
    "#-------------------------prepare to train--------------------------------------------\n",
    "def run_cmd(cmd):\n",
    "    print(cmd)\n",
    "    process = subprocess.Popen(cmd, shell=True,\n",
    "                       stdout=subprocess.PIPE,\n",
    "                        stderr=subprocess.STDOUT)\n",
    "    for t, line in enumerate(iter(process.stdout.readline,b'')):\n",
    "        line = line.decode('utf8').rstrip()\n",
    "        print(line)\n",
    "    process.communicate()\n",
    "    return process.returncode\n",
    "\n",
    "SentimentDocument = namedtuple('SentimentDocument', 'words tags')\n",
    "class Doc_list(object):\n",
    "    def __init__(self,f):\n",
    "        self.f = f\n",
    "    def __iter__(self):\n",
    "        for i,line in enumerate(codecs.open(self.f,encoding='utf8')):\n",
    "            words = line.split()\n",
    "            tags = [int(words[0][2:])]\n",
    "            words = words[1:]\n",
    "            yield SentimentDocument(words,tags)\n",
    "d2v = Doc2Vec(dm=0, size=300, negative=5, hs=0, min_count=3, window=30,sample=1e-5,workers=8,alpha=0.025,min_alpha=0.025)\n",
    "doc_list = Doc_list('alldata-id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "#-------------------train dbow doc2vec---------------------------------------------\n",
    "df_lb = pd.read_csv(cfg.data_path + 'all_v2.csv',usecols=['Education','age','gender'],nrows=100000)\n",
    "ys = {}\n",
    "for lb in ['Education','age','gender']:\n",
    "    ys[lb] = np.array(df_lb[lb])\n",
    "\n",
    "for i in range(2):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "#     run_cmd('shuf alldata-id.txt > alldata-id-shuf.txt')\n",
    "    doc_list = Doc_list('alldata-id.txt')\n",
    "    d2v.train(doc_list)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(100000)])\n",
    "    for lb in [\"Education\",'age','gender']:\n",
    "        scores = cross_val_score(LogisticRegression(C=3),X_d2v,ys[lb],cv=5)\n",
    "        print('dbow',lb,scores,np.mean(scores))\n",
    "d2v.save(cfg.data_path + 'dbow_d2v.model')\n",
    "print(datetime.now(),'save done')\n",
    "\n",
    "d2v = Doc2Vec(dm=1, size=300, negative=5, hs=0, min_count=3, window=10,sample=1e-5,workers=8,alpha=0.05,min_alpha=0.025)\n",
    "doc_list = Doc_list('alldata-id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "#---------------train dm doc2vec-----------------------------------------------------\n",
    "for i in range(10):\n",
    "    print(datetime.now(),'pass:',i)\n",
    "#     run_cmd('shuf alldata-id.txt > alldata-id-shuf.txt')\n",
    "    doc_list = Doc_list('alldata-id.txt')\n",
    "    d2v.train(doc_list)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(100000)])\n",
    "    for lb in [\"Education\",'age','gender']:\n",
    "        scores = cross_val_score(LogisticRegression(C=3),X_d2v,ys[lb],cv=5)\n",
    "        print('dm',lb,scores,np.mean(scores))\n",
    "d2v.save(cfg.data_path + 'dm_d2v.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000\n",
      "20000\n",
      "30000\n",
      "40000\n",
      "50000\n",
      "60000\n",
      "不灭剑祖\t六间房\t南艺新生颜值爆表\t阴模\t河南现纸片楼\t苹果手机定位追踪\t昔日港姐曝黑幕\t大主宰\t武神空间\t库克抛售苹果股份\t女演员徐婷病逝\t熊孩甜馨自己理发\t王宝强走出阴霾\t剑道通神txt下载\t张翰家门长蘑菇\t普京专车发生事故\t法师真理\t永恒剑主\t宁波二\t万古仙穹\t三蛮里的小六子\t透支9元还9000\t为女儿造血生3子\t朝鲜疑发生核爆\t从容淡定的某人\t雄霸蛮荒\t阿根廷捕获奇特鱼\t玄界之门\t雪鹰领主\t男子扛妻遗体步行\t不朽凡人\t六间房秀场直播大厅\t美又收天价核潜艇\t滨崎步宣布离婚\t商物模特\t凡人寻道\t陈乔恩遭网友炮轰\t钓鱼视频\t万域之王\t日本进入多死社会\t世界第一高桥合龙\t三毛里的小六子\t菲总统向中国求情\t天气预报\t台风把飞机吹跑\t老虎咬人调查结果\t武破万古\t武林半侠传\t买婚房却买到凶宅\t李多海承认恋情\t刘翔被曝求婚成功\t人道至尊\t渔民捕获怪头鱼\t出租绕道罚2000\t马蓉秒删微博\t刘子歌将嫁恩师\t福建土豪办婚礼\t女生求婚大爷\t飞豹战机赴俄参赛\t造化之王\t瑞士战机失踪\t麻雀收视火爆\t马云报仇买肯德基\t观众体验残奥项目\t剑道通神笔趣阁\t神藏\t剑道通神\t王宝强财产分割\t神马电影\t武炼巅峰\t香港慰问驻港部队\t一念永恒\t永恒之心\t男子术后双肾全无\t面部移植者辞世\t曼联计划贱卖鲁尼\t青岛对虾论只卖\t超凡传\t世界最美杀马特\t美国纽约发生爆炸\t巴基斯坦捐赠大米\t商务模特\t女司机草根遮号牌\t女孩放生六千头羊\t普京送安倍武士刀\t真武世界\t神\t朝鲜代表归国遇冷\t眼镜王蛇占屋为王\t连撞三车称遇到鬼\t凡人修仙\t傅园慧收贝尔战书\t宋小宝耍大牌被殴\t德国难民街头斗殴\t古力娜扎关闭评论\t巫界之树\t黄渤携妻女会友\t厦门重启限购\t剑游太虚\t美印签署后勤协定\t永恒圣帝\t纪晓岚和珅重聚\t剑主苍穹\t大话3遭吐槽\t炸鸡加罂粟壳判刑\t柯震东李毓芬复合\t龙符\n",
      "\n",
      "====================\n",
      "['不灭', '剑祖', '不灭_*_剑祖', '六间房', '南艺', '新生', '颜值', '爆表', '南艺_*_新生', '新生_*_颜值', '颜值_*_爆表', '阴模', '河南', '现', '纸片', '楼', '河南_*_现', '现_*_纸片', '纸片_*_楼', '苹果', '手机', '定位', '追踪', '苹果_*_手机', '手机_*_定位', '定位_*_追踪', '昔日', '港姐', '曝', '黑幕', '昔日_*_港姐', '港姐_*_曝', '曝_*_黑幕', '大', '主宰', '大_*_主宰', '武神', '空间', '武神_*_空间', '库克', '抛售', '苹果', '股份', '库克_*_抛售', '抛售_*_苹果', '苹果_*_股份', '女演员', '徐婷', '病逝', '女演员_*_徐婷', '徐婷_*_病逝', '熊孩', '甜馨', '自己', '理发', '熊孩_*_甜馨', '甜馨_*_自己', '自己_*_理发', '王宝强', '走出', '阴霾', '王宝强_*_走出', '走出_*_阴霾', '剑道', '通神', 'txt', '下载', '剑道_*_通神', '通神_*_txt', 'txt_*_下载', '张翰', '家门', '长', '蘑菇', '张翰_*_家门', '家门_*_长', '长_*_蘑菇', '普京', '专车', '发生', '事故', '普京_*_专车', '专车_*_发生', '发生_*_事故', '法师', '真理', '法师_*_真理', '永恒', '剑主', '永恒_*_剑主', '宁波', '二', '宁波_*_二', '万古', '仙穹', '万古_*_仙穹', '三蛮', '里', '的', '小六子', '三蛮_*_里', '里_*_的', '的_*_小六子', '透支', '9', '元', '还', '9000', '透支_*_9', '9_*_元', '元_*_还', '还_*_9000', '为', '女儿', '造血', '生', '3', '子', '为_*_女儿', '女儿_*_造血', '造血_*_生', '生_*_3', '3_*_子', '朝鲜', '疑', '发生', '核爆', '朝鲜_*_疑', '疑_*_发生', '发生_*_核爆', '从容', '淡定', '的', '某人', '从容_*_淡定', '淡定_*_的', '的_*_某人', '雄霸', '蛮荒', '雄霸_*_蛮荒', '阿根廷', '捕获', '奇特', '鱼', '阿根廷_*_捕获', '捕获_*_奇特', '奇特_*_鱼', '玄界', '之门', '玄界_*_之门', '雪鹰', '领主', '雪鹰_*_领主', '男子', '扛', '妻', '遗体', '步行', '男子_*_扛', '扛_*_妻', '妻_*_遗体', '遗体_*_步行', '不朽', '凡人', '不朽_*_凡人', '六间房', '秀场', '直播', '大厅', '六间房_*_秀场', '秀场_*_直播', '直播_*_大厅', '美', '又', '收', '天价', '核潜艇', '美_*_又', '又_*_收', '收_*_天价', '天价_*_核潜艇', '滨崎步', '宣布', '离婚', '滨崎步_*_宣布', '宣布_*_离婚', '商物', '模特', '商物_*_模特', '凡人', '寻', '道', '凡人_*_寻', '寻_*_道', '陈乔恩', '遭', '网友', '炮轰', '陈乔恩_*_遭', '遭_*_网友', '网友_*_炮轰', '钓鱼', '视频', '钓鱼_*_视频', '万域', '之王', '万域_*_之王', '日本', '进入', '多死', '社会', '日本_*_进入', '进入_*_多死', '多死_*_社会', '世界', '第一', '高桥', '合龙', '世界_*_第一', '第一_*_高桥', '高桥_*_合龙', '三毛', '里', '的', '小六子', '三毛_*_里', '里_*_的', '的_*_小六子', '菲', '总统', '向', '中国', '求情', '菲_*_总统', '总统_*_向', '向_*_中国', '中国_*_求情', '天气预报', '台风', '把', '飞机', '吹', '跑', '台风_*_把', '把_*_飞机', '飞机_*_吹', '吹_*_跑', '老虎', '咬人', '调查结果', '老虎_*_咬人', '咬人_*_调查结果', '武破', '万古', '武破_*_万古', '武林', '半侠', '传', '武林_*_半侠', '半侠_*_传', '买', '婚房', '却', '买', '到', '凶宅', '买_*_婚房', '婚房_*_却', '却_*_买', '买_*_到', '到_*_凶宅', '李多海', '承认', '恋情', '李多海_*_承认', '承认_*_恋情', '刘翔', '被', '曝', '求婚', '成功', '刘翔_*_被', '被_*_曝', '曝_*_求婚', '求婚_*_成功', '人', '道', '至尊', '人_*_道', '道_*_至尊', '渔民', '捕获', '怪头', '鱼', '渔民_*_捕获', '捕获_*_怪头', '怪头_*_鱼', '出租', '绕道', '罚', '2000', '出租_*_绕道', '绕道_*_罚', '罚_*_2000', '马蓉', '秒', '删微博', '马蓉_*_秒', '秒_*_删微博', '刘子', '歌', '将', '嫁', '恩师', '刘子_*_歌', '歌_*_将', '将_*_嫁', '嫁_*_恩师', '福建', '土豪', '办', '婚礼', '福建_*_土豪', '土豪_*_办', '办_*_婚礼', '女生', '求婚', '大爷', '女生_*_求婚', '求婚_*_大爷', '飞豹', '战机', '赴', '俄', '参赛', '飞豹_*_战机', '战机_*_赴', '赴_*_俄', '俄_*_参赛', '造化', '之王', '造化_*_之王', '瑞士', '战机', '失踪', '瑞士_*_战机', '战机_*_失踪', '麻雀', '收视', '火爆', '麻雀_*_收视', '收视_*_火爆', '马云', '报仇', '买', '肯德基', '马云_*_报仇', '报仇_*_买', '买_*_肯德基', '观众', '体验', '残奥', '项目', '观众_*_体验', '体验_*_残奥', '残奥_*_项目', '剑', '道', '通', '神笔', '趣阁', '剑_*_道', '道_*_通', '通_*_神笔', '神笔_*_趣阁', '神藏', '剑道', '通神', '剑道_*_通神', '王宝强', '财产', '分割', '王宝强_*_财产', '财产_*_分割', '神马', '电影', '神马_*_电影', '武炼', '巅峰', '武炼_*_巅峰', '香港', '慰问', '驻港部队', '香港_*_慰问', '慰问_*_驻港部队', '一', '念', '永恒', '一_*_念', '念_*_永恒', '永恒', '之心', '永恒_*_之心', '男子', '术后', '双肾', '全无', '男子_*_术后', '术后_*_双肾', '双肾_*_全无', '面部', '移植', '者', '辞世', '面部_*_移植', '移植_*_者', '者_*_辞世', '曼联', '计划', '贱卖', '鲁尼', '曼联_*_计划', '计划_*_贱卖', '贱卖_*_鲁尼', '青岛', '对虾', '论', '只', '卖', '青岛_*_对虾', '对虾_*_论', '论_*_只', '只_*_卖', '超凡', '传', '超凡_*_传', '世界', '最美', '杀马特', '世界_*_最美', '最美_*_杀马特', '美国纽约', '发生爆炸', '美国纽约_*_发生爆炸', '巴基斯坦', '捐赠', '大米', '巴基斯坦_*_捐赠', '捐赠_*_大米', '商务', '模特', '商务_*_模特', '女司机', '草根', '遮', '号牌', '女司机_*_草根', '草根_*_遮', '遮_*_号牌', '女孩', '放生', '六千', '头羊', '女孩_*_放生', '放生_*_六千', '六千_*_头羊', '普京', '送', '安倍', '武士刀', '普京_*_送', '送_*_安倍', '安倍_*_武士刀', '真武', '世界', '真武_*_世界', '神', '朝鲜', '代表', '归国', '遇冷', '朝鲜_*_代表', '代表_*_归国', '归国_*_遇冷', '眼镜', '王蛇', '占', '屋为', '王', '眼镜_*_王蛇', '王蛇_*_占', '占_*_屋为', '屋为_*_王', '连撞', '三车', '称', '遇到', '鬼', '连撞_*_三车', '三车_*_称', '称_*_遇到', '遇到_*_鬼', '凡人', '修仙', '凡人_*_修仙', '傅园', '慧收', '贝尔', '战书', '傅园_*_慧收', '慧收_*_贝尔', '贝尔_*_战书', '宋', '小宝', '耍大牌', '被', '殴', '宋_*_小宝', '小宝_*_耍大牌', '耍大牌_*_被', '被_*_殴', '德国', '难民', '街头', '斗殴', '德国_*_难民', '难民_*_街头', '街头_*_斗殴', '古力', '娜', '扎', '关闭', '评论', '古力_*_娜', '娜_*_扎', '扎_*_关闭', '关闭_*_评论', '巫界', '之树', '巫界_*_之树', '黄渤携', '妻女', '会友', '黄渤携_*_妻女', '妻女_*_会友', '厦门', '重启', '限购', '厦门_*_重启', '重启_*_限购', '剑游', '太虚', '剑游_*_太虚', '美印', '签署', '后勤', '协定', '美印_*_签署', '签署_*_后勤', '后勤_*_协定', '永恒', '圣帝', '永恒_*_圣帝', '纪晓岚', '和珅', '重聚', '纪晓岚_*_和珅', '和珅_*_重聚', '剑主', '苍穹', '剑主_*_苍穹', '大话', '3', '遭吐槽', '大话_*_3', '3_*_遭吐槽', '炸鸡', '加', '罂粟壳', '判刑', '炸鸡_*_加', '加_*_罂粟壳', '罂粟壳_*_判刑', '柯震东', '李毓芬', '复合', '柯震东_*_李毓芬', '李毓芬_*_复合', '龙符', '\\r\\n', '龙符_*_\\r\\n']\n",
      "70000\n",
      "80000\n",
      "90000\n",
      "100000\n",
      "Education\n",
      "2017-01-12 14:50:16.304664 stack:1/5\n",
      "va acc: 0.661875\n",
      "te acc: 0.65985\n",
      "2017-01-12 14:52:14.165609 stack:2/5\n",
      "va acc: 0.6548125\n",
      "te acc: 0.6585\n",
      "2017-01-12 14:54:19.879218 stack:3/5\n",
      "va acc: 0.653375\n",
      "te acc: 0.65795\n",
      "2017-01-12 14:56:09.618296 stack:4/5\n",
      "va acc: 0.655125\n",
      "te acc: 0.65925\n",
      "2017-01-12 14:57:53.542699 stack:5/5\n",
      "va acc: 0.65625\n",
      "te acc: 0.66155\n",
      "age\n",
      "2017-01-12 14:59:42.747756 stack:1/5\n",
      "va acc: 0.6036875\n",
      "te acc: 0.6048\n",
      "2017-01-12 15:01:45.330700 stack:2/5\n",
      "va acc: 0.5939375\n",
      "te acc: 0.60335\n",
      "2017-01-12 15:03:46.256206 stack:3/5\n",
      "va acc: 0.600125\n",
      "te acc: 0.6022\n",
      "2017-01-12 15:05:45.507223 stack:4/5\n",
      "va acc: 0.5996875\n",
      "te acc: 0.6024\n",
      "2017-01-12 15:07:52.540944 stack:5/5\n",
      "va acc: 0.6035\n",
      "te acc: 0.60365\n",
      "gender\n",
      "2017-01-12 15:10:04.738982 stack:1/5\n",
      "va acc: 0.83375\n",
      "te acc: 0.8307\n",
      "2017-01-12 15:10:24.870481 stack:2/5\n",
      "va acc: 0.8334375\n",
      "te acc: 0.83375\n",
      "2017-01-12 15:10:43.265079 stack:3/5\n",
      "va acc: 0.8304375\n",
      "te acc: 0.83335\n",
      "2017-01-12 15:11:01.031939 stack:4/5\n",
      "va acc: 0.834\n",
      "te acc: 0.83275\n",
      "2017-01-12 15:11:18.825272 stack:5/5\n",
      "va acc: 0.837875\n",
      "te acc: 0.8316\n",
      "2017-01-12 15:11:39.851072 save tfidf stack done!\n"
     ]
    }
   ],
   "source": [
    "'''tfidf-lr stack for education/age/gender'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import jieba\n",
    "import pickle\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cross_validation import KFold\n",
    "from datetime import datetime\n",
    "import cfg\n",
    "\n",
    "#-----------------------myfunc-----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "#-----------------------load data--------------------\n",
    "\n",
    "df_all = pd.read_csv(cfg.data_path + 'all_v2.csv',encoding='utf8',nrows=100000)\n",
    "ys = {}\n",
    "for label in ['Education','age','gender']:\n",
    "    ys[label] = np.array(df_all[label])\n",
    "\n",
    "class Tokenizer():\n",
    "    def __init__(self):\n",
    "        self.n = 0\n",
    "    def __call__(self,line):\n",
    "        tokens = []\n",
    "        for query in line.split('\\t'):\n",
    "            words = [word for word in jieba.cut(query)]\n",
    "            for gram in [1,2]:\n",
    "                for i in range(len(words) - gram + 1):\n",
    "                    tokens += [\"_*_\".join(words[i:i+gram])]\n",
    "        if np.random.rand() < 0.00001:\n",
    "            print(line)\n",
    "            print('='*20)\n",
    "            print(tokens)\n",
    "        self.n += 1\n",
    "        if self.n%10000==0:\n",
    "            print(self.n)\n",
    "        return tokens\n",
    "\n",
    "tfv = TfidfVectorizer(tokenizer=Tokenizer(),min_df=3,max_df=0.95,sublinear_tf=True)\n",
    "X_sp = tfv.fit_transform(df_all['query'])\n",
    "pickle.dump(X_sp,open(cfg.data_path + 'tfidf_10W.feat','wb'))\n",
    "\n",
    "df_stack = pd.DataFrame(index=range(len(df_all)))\n",
    "\n",
    "#-----------------------stack for education/age/gender------------------\n",
    "for lb in ['Education','age','gender']:\n",
    "    print(lb)\n",
    "    TR = 80000\n",
    "    num_class = len(pd.value_counts(ys[lb]))\n",
    "    n = 5\n",
    "\n",
    "    X = X_sp[:TR]\n",
    "    y = ys[lb][:TR]\n",
    "    X_te = X_sp[TR:]\n",
    "    y_te = ys[lb][TR:]\n",
    "\n",
    "    stack = np.zeros((X.shape[0],num_class))\n",
    "    stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "\n",
    "    for i,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "        print('%s stack:%d/%d'%(str(datetime.now()),i+1,n))\n",
    "        clf = LogisticRegression(C=3)\n",
    "        clf.fit(X[tr],y[tr])\n",
    "        y_pred_va = clf.predict_proba(X[va])\n",
    "        y_pred_te = clf.predict_proba(X_te)\n",
    "        print('va acc:',myAcc(y[va],y_pred_va))\n",
    "        print('te acc:',myAcc(y_te,y_pred_te))\n",
    "        stack[va] += y_pred_va\n",
    "        stack_te += y_pred_te\n",
    "    stack_te /= n\n",
    "    stack_all = np.vstack([stack,stack_te])\n",
    "    for i in range(stack_all.shape[1]):\n",
    "        df_stack['tfidf_{}_{}'.format(lb,i)] = stack_all[:,i]\n",
    "\n",
    "df_stack.to_csv(cfg.data_path + 'tfidf_stack_10W.csv',index=None,encoding='utf8')\n",
    "print(datetime.now(),'save tfidf stack done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:855: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n",
      "Using Theano backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-01-12 20:25:22.517795 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.9891 - acc: 0.6110 - val_loss: 0.9312 - val_acc: 0.6365\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9429 - acc: 0.6302 - val_loss: 0.9321 - val_acc: 0.6370\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9359 - acc: 0.6340 - val_loss: 0.9285 - val_acc: 0.6374\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9327 - acc: 0.6346 - val_loss: 0.9272 - val_acc: 0.6372\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9298 - acc: 0.6359 - val_loss: 0.9246 - val_acc: 0.6384\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9249 - acc: 0.6383 - val_loss: 0.9243 - val_acc: 0.6413\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9224 - acc: 0.6387 - val_loss: 0.9264 - val_acc: 0.6401\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9195 - acc: 0.6403 - val_loss: 0.9212 - val_acc: 0.6415\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9155 - acc: 0.6419 - val_loss: 0.9187 - val_acc: 0.6427\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9121 - acc: 0.6416 - val_loss: 0.9164 - val_acc: 0.6443\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9084 - acc: 0.6442 - val_loss: 0.9163 - val_acc: 0.6425\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9060 - acc: 0.6457 - val_loss: 0.9116 - val_acc: 0.6445\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9019 - acc: 0.6470 - val_loss: 0.9129 - val_acc: 0.6469\n",
      "Epoch 14/35\n",
      "2s - loss: 0.8996 - acc: 0.6477 - val_loss: 0.9141 - val_acc: 0.6432\n",
      "Epoch 15/35\n",
      "2s - loss: 0.8973 - acc: 0.6483 - val_loss: 0.9094 - val_acc: 0.6472\n",
      "Epoch 16/35\n",
      "2s - loss: 0.8942 - acc: 0.6494 - val_loss: 0.9081 - val_acc: 0.6475\n",
      "Epoch 17/35\n",
      "2s - loss: 0.8911 - acc: 0.6498 - val_loss: 0.9080 - val_acc: 0.6496\n",
      "Epoch 18/35\n",
      "2s - loss: 0.8884 - acc: 0.6508 - val_loss: 0.9046 - val_acc: 0.6499\n",
      "Epoch 19/35\n",
      "2s - loss: 0.8864 - acc: 0.6533 - val_loss: 0.9038 - val_acc: 0.6502\n",
      "Epoch 20/35\n",
      "2s - loss: 0.8825 - acc: 0.6544 - val_loss: 0.9041 - val_acc: 0.6503\n",
      "Epoch 21/35\n",
      "2s - loss: 0.8804 - acc: 0.6548 - val_loss: 0.9029 - val_acc: 0.6508\n",
      "Epoch 22/35\n",
      "2s - loss: 0.8774 - acc: 0.6563 - val_loss: 0.9032 - val_acc: 0.6494\n",
      "Epoch 23/35\n",
      "2s - loss: 0.8748 - acc: 0.6567 - val_loss: 0.9030 - val_acc: 0.6519\n",
      "Epoch 24/35\n",
      "2s - loss: 0.8735 - acc: 0.6579 - val_loss: 0.9020 - val_acc: 0.6512\n",
      "Epoch 25/35\n",
      "2s - loss: 0.8707 - acc: 0.6598 - val_loss: 0.9058 - val_acc: 0.6534\n",
      "Epoch 26/35\n",
      "2s - loss: 0.8673 - acc: 0.6607 - val_loss: 0.9004 - val_acc: 0.6539\n",
      "Epoch 27/35\n",
      "2s - loss: 0.8664 - acc: 0.6620 - val_loss: 0.9034 - val_acc: 0.6490\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8624 - acc: 0.6630 - val_loss: 0.9003 - val_acc: 0.6507\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8609 - acc: 0.6637 - val_loss: 0.9026 - val_acc: 0.6524\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8581 - acc: 0.6637 - val_loss: 0.9047 - val_acc: 0.6525\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8559 - acc: 0.6651 - val_loss: 0.9004 - val_acc: 0.6518\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8543 - acc: 0.6673 - val_loss: 0.9006 - val_acc: 0.6514\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8513 - acc: 0.6657 - val_loss: 0.9003 - val_acc: 0.6525\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8479 - acc: 0.6682 - val_loss: 0.9053 - val_acc: 0.6518\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8445 - acc: 0.6695 - val_loss: 0.9029 - val_acc: 0.6502\n",
      "19968/20000 [============================>.] - ETA: 0sva acc: 0.651\n",
      "te acc: 0.6502\n",
      "2017-01-12 20:27:41.823707 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.9836 - acc: 0.6117 - val_loss: 0.9314 - val_acc: 0.6354\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9362 - acc: 0.6327 - val_loss: 0.9307 - val_acc: 0.6393\n",
      "Epoch 3/35\n",
      "3s - loss: 0.9316 - acc: 0.6348 - val_loss: 0.9286 - val_acc: 0.6391\n",
      "Epoch 4/35\n",
      "3s - loss: 0.9280 - acc: 0.6354 - val_loss: 0.9279 - val_acc: 0.6382\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9245 - acc: 0.6359 - val_loss: 0.9282 - val_acc: 0.6391\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9217 - acc: 0.6386 - val_loss: 0.9236 - val_acc: 0.6413\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9185 - acc: 0.6387 - val_loss: 0.9240 - val_acc: 0.6405\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9146 - acc: 0.6398 - val_loss: 0.9214 - val_acc: 0.6431\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9113 - acc: 0.6413 - val_loss: 0.9195 - val_acc: 0.6411\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9083 - acc: 0.6411 - val_loss: 0.9201 - val_acc: 0.6431\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9046 - acc: 0.6431 - val_loss: 0.9156 - val_acc: 0.6435\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9012 - acc: 0.6452 - val_loss: 0.9139 - val_acc: 0.6442\n",
      "Epoch 13/35\n",
      "2s - loss: 0.8979 - acc: 0.6464 - val_loss: 0.9130 - val_acc: 0.6470\n",
      "Epoch 14/35\n",
      "2s - loss: 0.8952 - acc: 0.6479 - val_loss: 0.9114 - val_acc: 0.6484\n",
      "Epoch 15/35\n",
      "2s - loss: 0.8911 - acc: 0.6499 - val_loss: 0.9103 - val_acc: 0.6467\n",
      "Epoch 16/35\n",
      "2s - loss: 0.8889 - acc: 0.6509 - val_loss: 0.9089 - val_acc: 0.6492\n",
      "Epoch 17/35\n",
      "2s - loss: 0.8867 - acc: 0.6524 - val_loss: 0.9092 - val_acc: 0.6493\n",
      "Epoch 18/35\n",
      "2s - loss: 0.8852 - acc: 0.6526 - val_loss: 0.9074 - val_acc: 0.6486\n",
      "Epoch 19/35\n",
      "2s - loss: 0.8818 - acc: 0.6528 - val_loss: 0.9071 - val_acc: 0.6491\n",
      "Epoch 20/35\n",
      "2s - loss: 0.8801 - acc: 0.6542 - val_loss: 0.9066 - val_acc: 0.6489\n",
      "Epoch 21/35\n",
      "2s - loss: 0.8776 - acc: 0.6561 - val_loss: 0.9063 - val_acc: 0.6489\n",
      "Epoch 22/35\n",
      "2s - loss: 0.8751 - acc: 0.6574 - val_loss: 0.9054 - val_acc: 0.6475\n",
      "Epoch 23/35\n",
      "3s - loss: 0.8726 - acc: 0.6571 - val_loss: 0.9053 - val_acc: 0.6509\n",
      "Epoch 24/35\n",
      "3s - loss: 0.8706 - acc: 0.6580 - val_loss: 0.9078 - val_acc: 0.6514\n",
      "Epoch 25/35\n",
      "2s - loss: 0.8681 - acc: 0.6590 - val_loss: 0.9046 - val_acc: 0.6507\n",
      "Epoch 26/35\n",
      "2s - loss: 0.8648 - acc: 0.6602 - val_loss: 0.9051 - val_acc: 0.6520\n",
      "Epoch 27/35\n",
      "2s - loss: 0.8628 - acc: 0.6623 - val_loss: 0.9074 - val_acc: 0.6522\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8606 - acc: 0.6630 - val_loss: 0.9059 - val_acc: 0.6493\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8579 - acc: 0.6631 - val_loss: 0.9046 - val_acc: 0.6515\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8550 - acc: 0.6648 - val_loss: 0.9033 - val_acc: 0.6536\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8533 - acc: 0.6657 - val_loss: 0.9035 - val_acc: 0.6530\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8491 - acc: 0.6670 - val_loss: 0.9048 - val_acc: 0.6513\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8464 - acc: 0.6684 - val_loss: 0.9080 - val_acc: 0.6488\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8454 - acc: 0.6689 - val_loss: 0.9049 - val_acc: 0.6512\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8416 - acc: 0.6704 - val_loss: 0.9062 - val_acc: 0.6542\n",
      "19776/20000 [============================>.] - ETA: 0sva acc: 0.64575\n",
      "te acc: 0.65415\n",
      "2017-01-12 20:29:23.290720 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.9876 - acc: 0.6112 - val_loss: 0.9341 - val_acc: 0.6329\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9383 - acc: 0.6333 - val_loss: 0.9304 - val_acc: 0.6340\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9326 - acc: 0.6357 - val_loss: 0.9299 - val_acc: 0.6345\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9290 - acc: 0.6364 - val_loss: 0.9272 - val_acc: 0.6373\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9259 - acc: 0.6374 - val_loss: 0.9261 - val_acc: 0.6395\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9221 - acc: 0.6407 - val_loss: 0.9244 - val_acc: 0.6388\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9186 - acc: 0.6404 - val_loss: 0.9233 - val_acc: 0.6402\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9147 - acc: 0.6408 - val_loss: 0.9202 - val_acc: 0.6421\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9110 - acc: 0.6422 - val_loss: 0.9174 - val_acc: 0.6426\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9071 - acc: 0.6425 - val_loss: 0.9186 - val_acc: 0.6415\n",
      "Epoch 11/35\n",
      "3s - loss: 0.9050 - acc: 0.6449 - val_loss: 0.9184 - val_acc: 0.6449\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9013 - acc: 0.6456 - val_loss: 0.9117 - val_acc: 0.6471\n",
      "Epoch 13/35\n",
      "2s - loss: 0.8976 - acc: 0.6481 - val_loss: 0.9123 - val_acc: 0.6435\n",
      "Epoch 14/35\n",
      "3s - loss: 0.8952 - acc: 0.6503 - val_loss: 0.9106 - val_acc: 0.6480\n",
      "Epoch 15/35\n",
      "2s - loss: 0.8925 - acc: 0.6500 - val_loss: 0.9086 - val_acc: 0.6480\n",
      "Epoch 16/35\n",
      "2s - loss: 0.8905 - acc: 0.6488 - val_loss: 0.9091 - val_acc: 0.6475\n",
      "Epoch 17/35\n",
      "2s - loss: 0.8872 - acc: 0.6523 - val_loss: 0.9084 - val_acc: 0.6455\n",
      "Epoch 18/35\n",
      "2s - loss: 0.8839 - acc: 0.6526 - val_loss: 0.9060 - val_acc: 0.6472\n",
      "Epoch 19/35\n",
      "2s - loss: 0.8825 - acc: 0.6543 - val_loss: 0.9081 - val_acc: 0.6476\n",
      "Epoch 20/35\n",
      "2s - loss: 0.8799 - acc: 0.6553 - val_loss: 0.9052 - val_acc: 0.6487\n",
      "Epoch 21/35\n",
      "2s - loss: 0.8771 - acc: 0.6566 - val_loss: 0.9026 - val_acc: 0.6485\n",
      "Epoch 22/35\n",
      "2s - loss: 0.8750 - acc: 0.6571 - val_loss: 0.9047 - val_acc: 0.6512\n",
      "Epoch 23/35\n",
      "2s - loss: 0.8731 - acc: 0.6595 - val_loss: 0.9035 - val_acc: 0.6509\n",
      "Epoch 24/35\n",
      "2s - loss: 0.8706 - acc: 0.6600 - val_loss: 0.9047 - val_acc: 0.6490\n",
      "Epoch 25/35\n",
      "2s - loss: 0.8677 - acc: 0.6619 - val_loss: 0.9033 - val_acc: 0.6498\n",
      "Epoch 26/35\n",
      "2s - loss: 0.8664 - acc: 0.6618 - val_loss: 0.9064 - val_acc: 0.6509\n",
      "Epoch 27/35\n",
      "2s - loss: 0.8632 - acc: 0.6628 - val_loss: 0.9046 - val_acc: 0.6502\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8600 - acc: 0.6648 - val_loss: 0.9045 - val_acc: 0.6509\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8593 - acc: 0.6635 - val_loss: 0.9018 - val_acc: 0.6532\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8562 - acc: 0.6653 - val_loss: 0.9037 - val_acc: 0.6528\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8529 - acc: 0.6661 - val_loss: 0.9048 - val_acc: 0.6518\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8510 - acc: 0.6669 - val_loss: 0.9041 - val_acc: 0.6525\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8489 - acc: 0.6693 - val_loss: 0.9019 - val_acc: 0.6516\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8455 - acc: 0.6707 - val_loss: 0.9025 - val_acc: 0.6530\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8434 - acc: 0.6700 - val_loss: 0.9043 - val_acc: 0.6529\n",
      "19904/20000 [============================>.] - ETA: 0sva acc: 0.646375\n",
      "te acc: 0.6529\n",
      "2017-01-12 20:31:04.599969 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.9894 - acc: 0.6119 - val_loss: 0.9311 - val_acc: 0.6369\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9393 - acc: 0.6334 - val_loss: 0.9318 - val_acc: 0.6347\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9339 - acc: 0.6342 - val_loss: 0.9262 - val_acc: 0.6402\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9305 - acc: 0.6355 - val_loss: 0.9277 - val_acc: 0.6390\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9272 - acc: 0.6370 - val_loss: 0.9242 - val_acc: 0.6393\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9237 - acc: 0.6393 - val_loss: 0.9234 - val_acc: 0.6395\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9209 - acc: 0.6409 - val_loss: 0.9219 - val_acc: 0.6411\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9170 - acc: 0.6405 - val_loss: 0.9225 - val_acc: 0.6407\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9135 - acc: 0.6423 - val_loss: 0.9196 - val_acc: 0.6410\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9099 - acc: 0.6427 - val_loss: 0.9167 - val_acc: 0.6413\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9068 - acc: 0.6454 - val_loss: 0.9150 - val_acc: 0.6428\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9038 - acc: 0.6459 - val_loss: 0.9139 - val_acc: 0.6449\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9006 - acc: 0.6466 - val_loss: 0.9104 - val_acc: 0.6451\n",
      "Epoch 14/35\n",
      "2s - loss: 0.8968 - acc: 0.6486 - val_loss: 0.9125 - val_acc: 0.6434\n",
      "Epoch 15/35\n",
      "2s - loss: 0.8953 - acc: 0.6488 - val_loss: 0.9119 - val_acc: 0.6453\n",
      "Epoch 16/35\n",
      "2s - loss: 0.8920 - acc: 0.6508 - val_loss: 0.9067 - val_acc: 0.6482\n",
      "Epoch 17/35\n",
      "2s - loss: 0.8907 - acc: 0.6520 - val_loss: 0.9073 - val_acc: 0.6482\n",
      "Epoch 18/35\n",
      "2s - loss: 0.8865 - acc: 0.6536 - val_loss: 0.9069 - val_acc: 0.6484\n",
      "Epoch 19/35\n",
      "2s - loss: 0.8841 - acc: 0.6550 - val_loss: 0.9054 - val_acc: 0.6501\n",
      "Epoch 20/35\n",
      "2s - loss: 0.8819 - acc: 0.6550 - val_loss: 0.9060 - val_acc: 0.6511\n",
      "Epoch 21/35\n",
      "2s - loss: 0.8793 - acc: 0.6547 - val_loss: 0.9068 - val_acc: 0.6518\n",
      "Epoch 22/35\n",
      "2s - loss: 0.8770 - acc: 0.6572 - val_loss: 0.9041 - val_acc: 0.6515\n",
      "Epoch 23/35\n",
      "2s - loss: 0.8747 - acc: 0.6577 - val_loss: 0.9041 - val_acc: 0.6515\n",
      "Epoch 24/35\n",
      "2s - loss: 0.8717 - acc: 0.6570 - val_loss: 0.9039 - val_acc: 0.6531\n",
      "Epoch 25/35\n",
      "2s - loss: 0.8706 - acc: 0.6598 - val_loss: 0.9025 - val_acc: 0.6529\n",
      "Epoch 26/35\n",
      "2s - loss: 0.8679 - acc: 0.6608 - val_loss: 0.9017 - val_acc: 0.6527\n",
      "Epoch 27/35\n",
      "2s - loss: 0.8654 - acc: 0.6610 - val_loss: 0.9018 - val_acc: 0.6532\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8629 - acc: 0.6621 - val_loss: 0.9018 - val_acc: 0.6549\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8605 - acc: 0.6629 - val_loss: 0.9003 - val_acc: 0.6525\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8573 - acc: 0.6645 - val_loss: 0.9031 - val_acc: 0.6538\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8567 - acc: 0.6652 - val_loss: 0.9049 - val_acc: 0.6502\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8533 - acc: 0.6659 - val_loss: 0.9037 - val_acc: 0.6558\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8486 - acc: 0.6679 - val_loss: 0.9029 - val_acc: 0.6530\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8484 - acc: 0.6670 - val_loss: 0.9014 - val_acc: 0.6540\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8446 - acc: 0.6698 - val_loss: 0.9086 - val_acc: 0.6518\n",
      "19648/20000 [============================>.] - ETA: 0sva acc: 0.6474375\n",
      "te acc: 0.65175\n",
      "2017-01-12 20:32:43.006408 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.9903 - acc: 0.6103 - val_loss: 0.9320 - val_acc: 0.6345\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9400 - acc: 0.6312 - val_loss: 0.9295 - val_acc: 0.6361\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9342 - acc: 0.6338 - val_loss: 0.9287 - val_acc: 0.6380\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9310 - acc: 0.6351 - val_loss: 0.9273 - val_acc: 0.6400\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9275 - acc: 0.6351 - val_loss: 0.9277 - val_acc: 0.6385\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9239 - acc: 0.6386 - val_loss: 0.9238 - val_acc: 0.6372\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9211 - acc: 0.6378 - val_loss: 0.9240 - val_acc: 0.6414\n",
      "Epoch 8/35\n",
      "3s - loss: 0.9172 - acc: 0.6407 - val_loss: 0.9244 - val_acc: 0.6425\n",
      "Epoch 9/35\n",
      "3s - loss: 0.9141 - acc: 0.6401 - val_loss: 0.9175 - val_acc: 0.6429\n",
      "Epoch 10/35\n",
      "3s - loss: 0.9106 - acc: 0.6424 - val_loss: 0.9168 - val_acc: 0.6435\n",
      "Epoch 11/35\n",
      "3s - loss: 0.9077 - acc: 0.6422 - val_loss: 0.9153 - val_acc: 0.6418\n",
      "Epoch 12/35\n",
      "3s - loss: 0.9045 - acc: 0.6446 - val_loss: 0.9159 - val_acc: 0.6427\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9010 - acc: 0.6456 - val_loss: 0.9136 - val_acc: 0.6452\n",
      "Epoch 14/35\n",
      "3s - loss: 0.8972 - acc: 0.6486 - val_loss: 0.9122 - val_acc: 0.6475\n",
      "Epoch 15/35\n",
      "3s - loss: 0.8959 - acc: 0.6475 - val_loss: 0.9104 - val_acc: 0.6442\n",
      "Epoch 16/35\n",
      "3s - loss: 0.8929 - acc: 0.6505 - val_loss: 0.9107 - val_acc: 0.6442\n",
      "Epoch 17/35\n",
      "2s - loss: 0.8906 - acc: 0.6503 - val_loss: 0.9096 - val_acc: 0.6466\n",
      "Epoch 18/35\n",
      "2s - loss: 0.8887 - acc: 0.6524 - val_loss: 0.9086 - val_acc: 0.6454\n",
      "Epoch 19/35\n",
      "2s - loss: 0.8846 - acc: 0.6535 - val_loss: 0.9101 - val_acc: 0.6473\n",
      "Epoch 20/35\n",
      "2s - loss: 0.8837 - acc: 0.6524 - val_loss: 0.9078 - val_acc: 0.6492\n",
      "Epoch 21/35\n",
      "2s - loss: 0.8798 - acc: 0.6542 - val_loss: 0.9104 - val_acc: 0.6498\n",
      "Epoch 22/35\n",
      "2s - loss: 0.8785 - acc: 0.6547 - val_loss: 0.9080 - val_acc: 0.6502\n",
      "Epoch 23/35\n",
      "2s - loss: 0.8760 - acc: 0.6567 - val_loss: 0.9051 - val_acc: 0.6501\n",
      "Epoch 24/35\n",
      "2s - loss: 0.8738 - acc: 0.6586 - val_loss: 0.9099 - val_acc: 0.6489\n",
      "Epoch 25/35\n",
      "2s - loss: 0.8720 - acc: 0.6589 - val_loss: 0.9058 - val_acc: 0.6509\n",
      "Epoch 26/35\n",
      "2s - loss: 0.8688 - acc: 0.6595 - val_loss: 0.9045 - val_acc: 0.6512\n",
      "Epoch 27/35\n",
      "3s - loss: 0.8672 - acc: 0.6596 - val_loss: 0.9048 - val_acc: 0.6532\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8641 - acc: 0.6635 - val_loss: 0.9043 - val_acc: 0.6507\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8624 - acc: 0.6630 - val_loss: 0.9047 - val_acc: 0.6511\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8595 - acc: 0.6623 - val_loss: 0.9046 - val_acc: 0.6500\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8565 - acc: 0.6656 - val_loss: 0.9095 - val_acc: 0.6536\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8551 - acc: 0.6653 - val_loss: 0.9064 - val_acc: 0.6512\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8521 - acc: 0.6663 - val_loss: 0.9040 - val_acc: 0.6533\n",
      "Epoch 34/35\n",
      "3s - loss: 0.8500 - acc: 0.6681 - val_loss: 0.9032 - val_acc: 0.6531\n",
      "Epoch 35/35\n",
      "3s - loss: 0.8474 - acc: 0.6680 - val_loss: 0.9053 - val_acc: 0.6480\n",
      "19840/20000 [============================>.] - ETA: 0sva acc: 0.6441875\n",
      "te acc: 0.64795\n",
      "2017-01-12 20:34:26.417557 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1038 - acc: 0.5682 - val_loss: 1.0460 - val_acc: 0.5965\n",
      "Epoch 2/35\n",
      "3s - loss: 1.0546 - acc: 0.5906 - val_loss: 1.0396 - val_acc: 0.6012\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0485 - acc: 0.5922 - val_loss: 1.0383 - val_acc: 0.6017\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0450 - acc: 0.5934 - val_loss: 1.0391 - val_acc: 0.6011\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0413 - acc: 0.5940 - val_loss: 1.0399 - val_acc: 0.6025\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0367 - acc: 0.5959 - val_loss: 1.0330 - val_acc: 0.6039\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0327 - acc: 0.5977 - val_loss: 1.0306 - val_acc: 0.6048\n",
      "Epoch 8/35\n",
      "3s - loss: 1.0292 - acc: 0.6002 - val_loss: 1.0275 - val_acc: 0.6037\n",
      "Epoch 9/35\n",
      "3s - loss: 1.0244 - acc: 0.6020 - val_loss: 1.0275 - val_acc: 0.6060\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0193 - acc: 0.6026 - val_loss: 1.0230 - val_acc: 0.6060\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0160 - acc: 0.6043 - val_loss: 1.0193 - val_acc: 0.6071\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0125 - acc: 0.6050 - val_loss: 1.0210 - val_acc: 0.6089\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0083 - acc: 0.6074 - val_loss: 1.0158 - val_acc: 0.6080\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0055 - acc: 0.6083 - val_loss: 1.0141 - val_acc: 0.6104\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0008 - acc: 0.6091 - val_loss: 1.0122 - val_acc: 0.6111\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9989 - acc: 0.6096 - val_loss: 1.0117 - val_acc: 0.6106\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9961 - acc: 0.6121 - val_loss: 1.0102 - val_acc: 0.6113\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9937 - acc: 0.6131 - val_loss: 1.0105 - val_acc: 0.6127\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9907 - acc: 0.6145 - val_loss: 1.0096 - val_acc: 0.6130\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9866 - acc: 0.6158 - val_loss: 1.0090 - val_acc: 0.6102\n",
      "Epoch 21/35\n",
      "3s - loss: 0.9842 - acc: 0.6160 - val_loss: 1.0080 - val_acc: 0.6120\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9810 - acc: 0.6180 - val_loss: 1.0060 - val_acc: 0.6106\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9787 - acc: 0.6195 - val_loss: 1.0036 - val_acc: 0.6132\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9751 - acc: 0.6212 - val_loss: 1.0043 - val_acc: 0.6149\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9732 - acc: 0.6222 - val_loss: 1.0135 - val_acc: 0.6120\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9719 - acc: 0.6222 - val_loss: 1.0040 - val_acc: 0.6128\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9668 - acc: 0.6225 - val_loss: 1.0055 - val_acc: 0.6134\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9648 - acc: 0.6245 - val_loss: 1.0057 - val_acc: 0.6141\n",
      "Epoch 29/35\n",
      "2s - loss: 0.9620 - acc: 0.6276 - val_loss: 1.0071 - val_acc: 0.6121\n",
      "Epoch 30/35\n",
      "2s - loss: 0.9576 - acc: 0.6274 - val_loss: 1.0024 - val_acc: 0.6139\n",
      "Epoch 31/35\n",
      "3s - loss: 0.9548 - acc: 0.6304 - val_loss: 1.0054 - val_acc: 0.6169\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9518 - acc: 0.6302 - val_loss: 1.0037 - val_acc: 0.6135\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9485 - acc: 0.6320 - val_loss: 1.0051 - val_acc: 0.6140\n",
      "Epoch 34/35\n",
      "3s - loss: 0.9452 - acc: 0.6339 - val_loss: 1.0030 - val_acc: 0.6142\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9417 - acc: 0.6350 - val_loss: 1.0043 - val_acc: 0.6143\n",
      "16000/16000 [==============================] - 0s     \n",
      "19776/20000 [============================>.] - ETA: 0sva acc: 0.6075\n",
      "te acc: 0.6143\n",
      "2017-01-12 20:36:06.281297 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0954 - acc: 0.5721 - val_loss: 1.0442 - val_acc: 0.5960\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0504 - acc: 0.5912 - val_loss: 1.0407 - val_acc: 0.5975\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0445 - acc: 0.5937 - val_loss: 1.0401 - val_acc: 0.5981\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0400 - acc: 0.5958 - val_loss: 1.0409 - val_acc: 0.5963\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0361 - acc: 0.5972 - val_loss: 1.0380 - val_acc: 0.5990\n",
      "Epoch 6/35\n",
      "3s - loss: 1.0320 - acc: 0.5987 - val_loss: 1.0352 - val_acc: 0.5988\n",
      "Epoch 7/35\n",
      "3s - loss: 1.0275 - acc: 0.5987 - val_loss: 1.0307 - val_acc: 0.6008\n",
      "Epoch 8/35\n",
      "3s - loss: 1.0233 - acc: 0.6008 - val_loss: 1.0320 - val_acc: 0.6015\n",
      "Epoch 9/35\n",
      "3s - loss: 1.0174 - acc: 0.6046 - val_loss: 1.0258 - val_acc: 0.6039\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0140 - acc: 0.6038 - val_loss: 1.0265 - val_acc: 0.6029\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0103 - acc: 0.6063 - val_loss: 1.0214 - val_acc: 0.6061\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0069 - acc: 0.6070 - val_loss: 1.0207 - val_acc: 0.6072\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0038 - acc: 0.6095 - val_loss: 1.0178 - val_acc: 0.6065\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0009 - acc: 0.6096 - val_loss: 1.0212 - val_acc: 0.6069\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9974 - acc: 0.6117 - val_loss: 1.0160 - val_acc: 0.6078\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9933 - acc: 0.6126 - val_loss: 1.0139 - val_acc: 0.6076\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9915 - acc: 0.6155 - val_loss: 1.0148 - val_acc: 0.6118\n",
      "Epoch 18/35\n",
      "3s - loss: 0.9882 - acc: 0.6153 - val_loss: 1.0142 - val_acc: 0.6085\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9860 - acc: 0.6159 - val_loss: 1.0093 - val_acc: 0.6099\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9829 - acc: 0.6177 - val_loss: 1.0088 - val_acc: 0.6136\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9798 - acc: 0.6187 - val_loss: 1.0082 - val_acc: 0.6149\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9778 - acc: 0.6187 - val_loss: 1.0080 - val_acc: 0.6132\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9754 - acc: 0.6198 - val_loss: 1.0128 - val_acc: 0.6103\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9714 - acc: 0.6209 - val_loss: 1.0089 - val_acc: 0.6109\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9689 - acc: 0.6231 - val_loss: 1.0060 - val_acc: 0.6132\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9659 - acc: 0.6241 - val_loss: 1.0106 - val_acc: 0.6122\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9634 - acc: 0.6252 - val_loss: 1.0072 - val_acc: 0.6133\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9600 - acc: 0.6262 - val_loss: 1.0065 - val_acc: 0.6139\n",
      "Epoch 29/35\n",
      "2s - loss: 0.9568 - acc: 0.6282 - val_loss: 1.0048 - val_acc: 0.6138\n",
      "Epoch 30/35\n",
      "2s - loss: 0.9558 - acc: 0.6293 - val_loss: 1.0068 - val_acc: 0.6139\n",
      "Epoch 31/35\n",
      "2s - loss: 0.9518 - acc: 0.6298 - val_loss: 1.0031 - val_acc: 0.6130\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9487 - acc: 0.6314 - val_loss: 1.0049 - val_acc: 0.6132\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9462 - acc: 0.6327 - val_loss: 1.0079 - val_acc: 0.6124\n",
      "Epoch 34/35\n",
      "2s - loss: 0.9436 - acc: 0.6319 - val_loss: 1.0146 - val_acc: 0.6130\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9394 - acc: 0.6361 - val_loss: 1.0062 - val_acc: 0.6132\n",
      "19008/20000 [===========================>..] - ETA: 0sva acc: 0.601125\n",
      "te acc: 0.61325\n",
      "2017-01-12 20:37:48.582170 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1053 - acc: 0.5689 - val_loss: 1.0412 - val_acc: 0.6007\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0538 - acc: 0.5906 - val_loss: 1.0400 - val_acc: 0.6005\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0479 - acc: 0.5913 - val_loss: 1.0393 - val_acc: 0.5987\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0436 - acc: 0.5952 - val_loss: 1.0418 - val_acc: 0.5980\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0395 - acc: 0.5969 - val_loss: 1.0362 - val_acc: 0.6022\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0343 - acc: 0.5978 - val_loss: 1.0307 - val_acc: 0.6031\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0300 - acc: 0.5988 - val_loss: 1.0344 - val_acc: 0.6021\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0252 - acc: 0.6016 - val_loss: 1.0259 - val_acc: 0.6045\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0207 - acc: 0.6026 - val_loss: 1.0231 - val_acc: 0.6036\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0161 - acc: 0.6028 - val_loss: 1.0220 - val_acc: 0.6040\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0122 - acc: 0.6066 - val_loss: 1.0180 - val_acc: 0.6083\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0090 - acc: 0.6068 - val_loss: 1.0161 - val_acc: 0.6104\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0054 - acc: 0.6084 - val_loss: 1.0154 - val_acc: 0.6103\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0016 - acc: 0.6099 - val_loss: 1.0131 - val_acc: 0.6100\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9981 - acc: 0.6119 - val_loss: 1.0216 - val_acc: 0.6047\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9957 - acc: 0.6123 - val_loss: 1.0106 - val_acc: 0.6116\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9930 - acc: 0.6132 - val_loss: 1.0135 - val_acc: 0.6101\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9903 - acc: 0.6129 - val_loss: 1.0080 - val_acc: 0.6110\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9864 - acc: 0.6171 - val_loss: 1.0094 - val_acc: 0.6106\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9846 - acc: 0.6157 - val_loss: 1.0082 - val_acc: 0.6132\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9828 - acc: 0.6182 - val_loss: 1.0076 - val_acc: 0.6147\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9790 - acc: 0.6184 - val_loss: 1.0094 - val_acc: 0.6122\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9760 - acc: 0.6209 - val_loss: 1.0066 - val_acc: 0.6141\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9742 - acc: 0.6208 - val_loss: 1.0028 - val_acc: 0.6147\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9713 - acc: 0.6235 - val_loss: 1.0059 - val_acc: 0.6125\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9687 - acc: 0.6235 - val_loss: 1.0047 - val_acc: 0.6100\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9650 - acc: 0.6248 - val_loss: 1.0075 - val_acc: 0.6122\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9625 - acc: 0.6260 - val_loss: 1.0031 - val_acc: 0.6140\n",
      "Epoch 29/35\n",
      "2s - loss: 0.9596 - acc: 0.6275 - val_loss: 1.0031 - val_acc: 0.6129\n",
      "Epoch 30/35\n",
      "3s - loss: 0.9560 - acc: 0.6279 - val_loss: 1.0041 - val_acc: 0.6134\n",
      "Epoch 31/35\n",
      "3s - loss: 0.9556 - acc: 0.6296 - val_loss: 1.0061 - val_acc: 0.6132\n",
      "Epoch 32/35\n",
      "3s - loss: 0.9513 - acc: 0.6312 - val_loss: 1.0027 - val_acc: 0.6142\n",
      "Epoch 33/35\n",
      "3s - loss: 0.9483 - acc: 0.6318 - val_loss: 1.0054 - val_acc: 0.6133\n",
      "Epoch 34/35\n",
      "3s - loss: 0.9441 - acc: 0.6334 - val_loss: 1.0038 - val_acc: 0.6130\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9419 - acc: 0.6330 - val_loss: 1.0046 - val_acc: 0.6138\n",
      "19808/20000 [============================>.] - ETA: 0sva acc: 0.6089375\n",
      "te acc: 0.6138\n",
      "2017-01-12 20:39:29.849086 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1037 - acc: 0.5679 - val_loss: 1.0484 - val_acc: 0.5952\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0547 - acc: 0.5890 - val_loss: 1.0415 - val_acc: 0.5986\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0502 - acc: 0.5934 - val_loss: 1.0426 - val_acc: 0.5954\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0456 - acc: 0.5929 - val_loss: 1.0405 - val_acc: 0.5976\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0421 - acc: 0.5950 - val_loss: 1.0389 - val_acc: 0.5974\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0373 - acc: 0.5978 - val_loss: 1.0369 - val_acc: 0.6036\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0330 - acc: 0.5997 - val_loss: 1.0316 - val_acc: 0.6018\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0280 - acc: 0.5990 - val_loss: 1.0307 - val_acc: 0.6036\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0239 - acc: 0.6015 - val_loss: 1.0276 - val_acc: 0.6054\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0191 - acc: 0.6033 - val_loss: 1.0318 - val_acc: 0.6003\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0153 - acc: 0.6055 - val_loss: 1.0271 - val_acc: 0.6055\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0108 - acc: 0.6089 - val_loss: 1.0177 - val_acc: 0.6060\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0073 - acc: 0.6070 - val_loss: 1.0147 - val_acc: 0.6087\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0047 - acc: 0.6086 - val_loss: 1.0175 - val_acc: 0.6067\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0012 - acc: 0.6105 - val_loss: 1.0125 - val_acc: 0.6103\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9987 - acc: 0.6113 - val_loss: 1.0129 - val_acc: 0.6108\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9950 - acc: 0.6123 - val_loss: 1.0157 - val_acc: 0.6124\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9928 - acc: 0.6148 - val_loss: 1.0093 - val_acc: 0.6118\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9896 - acc: 0.6159 - val_loss: 1.0086 - val_acc: 0.6119\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9874 - acc: 0.6149 - val_loss: 1.0109 - val_acc: 0.6122\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9846 - acc: 0.6173 - val_loss: 1.0099 - val_acc: 0.6115\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9814 - acc: 0.6196 - val_loss: 1.0075 - val_acc: 0.6139\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9781 - acc: 0.6200 - val_loss: 1.0039 - val_acc: 0.6123\n",
      "Epoch 24/35\n",
      "3s - loss: 0.9767 - acc: 0.6224 - val_loss: 1.0067 - val_acc: 0.6126\n",
      "Epoch 25/35\n",
      "3s - loss: 0.9728 - acc: 0.6238 - val_loss: 1.0051 - val_acc: 0.6153\n",
      "Epoch 26/35\n",
      "3s - loss: 0.9716 - acc: 0.6232 - val_loss: 1.0043 - val_acc: 0.6154\n",
      "Epoch 27/35\n",
      "3s - loss: 0.9684 - acc: 0.6243 - val_loss: 1.0043 - val_acc: 0.6147\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9652 - acc: 0.6254 - val_loss: 1.0044 - val_acc: 0.6143\n",
      "Epoch 29/35\n",
      "3s - loss: 0.9622 - acc: 0.6266 - val_loss: 1.0033 - val_acc: 0.6149\n",
      "Epoch 30/35\n",
      "3s - loss: 0.9599 - acc: 0.6270 - val_loss: 1.0080 - val_acc: 0.6124\n",
      "Epoch 31/35\n",
      "3s - loss: 0.9567 - acc: 0.6286 - val_loss: 1.0035 - val_acc: 0.6137\n",
      "Epoch 32/35\n",
      "3s - loss: 0.9534 - acc: 0.6308 - val_loss: 1.0045 - val_acc: 0.6141\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9509 - acc: 0.6321 - val_loss: 1.0075 - val_acc: 0.6120\n",
      "Epoch 34/35\n",
      "3s - loss: 0.9493 - acc: 0.6316 - val_loss: 1.0040 - val_acc: 0.6132\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9441 - acc: 0.6345 - val_loss: 1.0036 - val_acc: 0.6169\n",
      "19904/20000 [============================>.] - ETA: 0sva acc: 0.604625\n",
      "te acc: 0.61685\n",
      "2017-01-12 20:41:12.806936 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "3s - loss: 1.1071 - acc: 0.5663 - val_loss: 1.0433 - val_acc: 0.5985\n",
      "Epoch 2/35\n",
      "3s - loss: 1.0565 - acc: 0.5883 - val_loss: 1.0390 - val_acc: 0.6008\n",
      "Epoch 3/35\n",
      "3s - loss: 1.0500 - acc: 0.5918 - val_loss: 1.0403 - val_acc: 0.5962\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0452 - acc: 0.5939 - val_loss: 1.0427 - val_acc: 0.5985\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0419 - acc: 0.5951 - val_loss: 1.0346 - val_acc: 0.6019\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0372 - acc: 0.5972 - val_loss: 1.0327 - val_acc: 0.6017\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0329 - acc: 0.5987 - val_loss: 1.0308 - val_acc: 0.6042\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0290 - acc: 0.5982 - val_loss: 1.0302 - val_acc: 0.6008\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0248 - acc: 0.6017 - val_loss: 1.0251 - val_acc: 0.6062\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0198 - acc: 0.6032 - val_loss: 1.0217 - val_acc: 0.6075\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0158 - acc: 0.6055 - val_loss: 1.0202 - val_acc: 0.6055\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0125 - acc: 0.6073 - val_loss: 1.0222 - val_acc: 0.6031\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0087 - acc: 0.6073 - val_loss: 1.0160 - val_acc: 0.6083\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0053 - acc: 0.6082 - val_loss: 1.0160 - val_acc: 0.6073\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0024 - acc: 0.6107 - val_loss: 1.0137 - val_acc: 0.6102\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9994 - acc: 0.6125 - val_loss: 1.0126 - val_acc: 0.6105\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9967 - acc: 0.6125 - val_loss: 1.0120 - val_acc: 0.6100\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9944 - acc: 0.6126 - val_loss: 1.0089 - val_acc: 0.6123\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9903 - acc: 0.6142 - val_loss: 1.0125 - val_acc: 0.6100\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9881 - acc: 0.6160 - val_loss: 1.0134 - val_acc: 0.6104\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9864 - acc: 0.6157 - val_loss: 1.0068 - val_acc: 0.6113\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9826 - acc: 0.6178 - val_loss: 1.0051 - val_acc: 0.6120\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9793 - acc: 0.6187 - val_loss: 1.0055 - val_acc: 0.6142\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9768 - acc: 0.6203 - val_loss: 1.0066 - val_acc: 0.6137\n",
      "Epoch 25/35\n",
      "3s - loss: 0.9760 - acc: 0.6208 - val_loss: 1.0054 - val_acc: 0.6148\n",
      "Epoch 26/35\n",
      "3s - loss: 0.9730 - acc: 0.6215 - val_loss: 1.0050 - val_acc: 0.6119\n",
      "Epoch 27/35\n",
      "3s - loss: 0.9698 - acc: 0.6222 - val_loss: 1.0034 - val_acc: 0.6133\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9666 - acc: 0.6253 - val_loss: 1.0041 - val_acc: 0.6129\n",
      "Epoch 29/35\n",
      "2s - loss: 0.9630 - acc: 0.6249 - val_loss: 1.0040 - val_acc: 0.6129\n",
      "Epoch 30/35\n",
      "2s - loss: 0.9620 - acc: 0.6268 - val_loss: 1.0032 - val_acc: 0.6144\n",
      "Epoch 31/35\n",
      "2s - loss: 0.9579 - acc: 0.6281 - val_loss: 1.0062 - val_acc: 0.6147\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9560 - acc: 0.6288 - val_loss: 1.0039 - val_acc: 0.6126\n",
      "Epoch 33/35\n",
      "3s - loss: 0.9518 - acc: 0.6320 - val_loss: 1.0119 - val_acc: 0.6127\n",
      "Epoch 34/35\n",
      "3s - loss: 0.9488 - acc: 0.6307 - val_loss: 1.0032 - val_acc: 0.6143\n",
      "Epoch 35/35\n",
      "3s - loss: 0.9453 - acc: 0.6353 - val_loss: 1.0040 - val_acc: 0.6120\n",
      "19648/20000 [============================>.] - ETA: 0sva acc: 0.6106875\n",
      "te acc: 0.61205\n",
      "2017-01-12 20:42:53.834745 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4406 - acc: 0.8240 - val_loss: 0.4273 - val_acc: 0.8369\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4271 - acc: 0.8347 - val_loss: 0.4256 - val_acc: 0.8369\n",
      "Epoch 3/35\n",
      "3s - loss: 0.4254 - acc: 0.8357 - val_loss: 0.4244 - val_acc: 0.8361\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4235 - acc: 0.8372 - val_loss: 0.4242 - val_acc: 0.8375\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4216 - acc: 0.8374 - val_loss: 0.4225 - val_acc: 0.8369\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4202 - acc: 0.8374 - val_loss: 0.4216 - val_acc: 0.8379\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4178 - acc: 0.8381 - val_loss: 0.4211 - val_acc: 0.8368\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4162 - acc: 0.8397 - val_loss: 0.4201 - val_acc: 0.8389\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4146 - acc: 0.8385 - val_loss: 0.4203 - val_acc: 0.8359\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4133 - acc: 0.8390 - val_loss: 0.4182 - val_acc: 0.8391\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4123 - acc: 0.8396 - val_loss: 0.4169 - val_acc: 0.8370\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4102 - acc: 0.8408 - val_loss: 0.4177 - val_acc: 0.8378\n",
      "Epoch 13/35\n",
      "3s - loss: 0.4097 - acc: 0.8413 - val_loss: 0.4157 - val_acc: 0.8368\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4082 - acc: 0.8409 - val_loss: 0.4141 - val_acc: 0.8402\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4065 - acc: 0.8414 - val_loss: 0.4141 - val_acc: 0.8390\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4051 - acc: 0.8419 - val_loss: 0.4134 - val_acc: 0.8371\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4037 - acc: 0.8423 - val_loss: 0.4114 - val_acc: 0.8404\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4025 - acc: 0.8433 - val_loss: 0.4130 - val_acc: 0.8417\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4004 - acc: 0.8435 - val_loss: 0.4128 - val_acc: 0.8432\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4004 - acc: 0.8441 - val_loss: 0.4097 - val_acc: 0.8423\n",
      "Epoch 21/35\n",
      "2s - loss: 0.3981 - acc: 0.8444 - val_loss: 0.4104 - val_acc: 0.8385\n",
      "Epoch 22/35\n",
      "2s - loss: 0.3970 - acc: 0.8444 - val_loss: 0.4089 - val_acc: 0.8400\n",
      "Epoch 23/35\n",
      "2s - loss: 0.3966 - acc: 0.8450 - val_loss: 0.4081 - val_acc: 0.8424\n",
      "Epoch 24/35\n",
      "3s - loss: 0.3952 - acc: 0.8455 - val_loss: 0.4125 - val_acc: 0.8374\n",
      "Epoch 25/35\n",
      "2s - loss: 0.3928 - acc: 0.8450 - val_loss: 0.4099 - val_acc: 0.8412\n",
      "Epoch 26/35\n",
      "2s - loss: 0.3929 - acc: 0.8456 - val_loss: 0.4094 - val_acc: 0.8425\n",
      "Epoch 27/35\n",
      "3s - loss: 0.3908 - acc: 0.8455 - val_loss: 0.4116 - val_acc: 0.8377\n",
      "Epoch 28/35\n",
      "2s - loss: 0.3893 - acc: 0.8464 - val_loss: 0.4086 - val_acc: 0.8415\n",
      "Epoch 29/35\n",
      "3s - loss: 0.3893 - acc: 0.8466 - val_loss: 0.4112 - val_acc: 0.8397\n",
      "Epoch 30/35\n",
      "2s - loss: 0.3872 - acc: 0.8478 - val_loss: 0.4083 - val_acc: 0.8395\n",
      "Epoch 31/35\n",
      "2s - loss: 0.3858 - acc: 0.8482 - val_loss: 0.4089 - val_acc: 0.8421\n",
      "Epoch 32/35\n",
      "2s - loss: 0.3846 - acc: 0.8482 - val_loss: 0.4108 - val_acc: 0.8425\n",
      "Epoch 33/35\n",
      "2s - loss: 0.3820 - acc: 0.8485 - val_loss: 0.4076 - val_acc: 0.8420\n",
      "Epoch 34/35\n",
      "2s - loss: 0.3825 - acc: 0.8487 - val_loss: 0.4100 - val_acc: 0.8430\n",
      "Epoch 35/35\n",
      "2s - loss: 0.3803 - acc: 0.8498 - val_loss: 0.4090 - val_acc: 0.8398\n",
      "19584/20000 [============================>.] - ETA: 0sva acc: 0.8420625\n",
      "te acc: 0.83985\n",
      "2017-01-12 20:44:34.398119 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4382 - acc: 0.8269 - val_loss: 0.4274 - val_acc: 0.8355\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4268 - acc: 0.8360 - val_loss: 0.4336 - val_acc: 0.8334\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4256 - acc: 0.8359 - val_loss: 0.4274 - val_acc: 0.8380\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4232 - acc: 0.8378 - val_loss: 0.4323 - val_acc: 0.8266\n",
      "Epoch 5/35\n",
      "3s - loss: 0.4217 - acc: 0.8381 - val_loss: 0.4243 - val_acc: 0.8353\n",
      "Epoch 6/35\n",
      "3s - loss: 0.4198 - acc: 0.8386 - val_loss: 0.4236 - val_acc: 0.8361\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4182 - acc: 0.8390 - val_loss: 0.4212 - val_acc: 0.8385\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4165 - acc: 0.8391 - val_loss: 0.4214 - val_acc: 0.8381\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4145 - acc: 0.8399 - val_loss: 0.4199 - val_acc: 0.8377\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4135 - acc: 0.8398 - val_loss: 0.4184 - val_acc: 0.8398\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4122 - acc: 0.8397 - val_loss: 0.4173 - val_acc: 0.8389\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4105 - acc: 0.8410 - val_loss: 0.4180 - val_acc: 0.8403\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4093 - acc: 0.8409 - val_loss: 0.4159 - val_acc: 0.8396\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4081 - acc: 0.8417 - val_loss: 0.4152 - val_acc: 0.8390\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4071 - acc: 0.8423 - val_loss: 0.4147 - val_acc: 0.8395\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4057 - acc: 0.8429 - val_loss: 0.4175 - val_acc: 0.8410\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4045 - acc: 0.8423 - val_loss: 0.4134 - val_acc: 0.8396\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4028 - acc: 0.8430 - val_loss: 0.4124 - val_acc: 0.8417\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4015 - acc: 0.8444 - val_loss: 0.4129 - val_acc: 0.8378\n",
      "Epoch 20/35\n",
      "2s - loss: 0.3998 - acc: 0.8443 - val_loss: 0.4099 - val_acc: 0.8414\n",
      "Epoch 21/35\n",
      "2s - loss: 0.3991 - acc: 0.8453 - val_loss: 0.4103 - val_acc: 0.8416\n",
      "Epoch 22/35\n",
      "2s - loss: 0.3971 - acc: 0.8452 - val_loss: 0.4105 - val_acc: 0.8395\n",
      "Epoch 23/35\n",
      "3s - loss: 0.3961 - acc: 0.8452 - val_loss: 0.4094 - val_acc: 0.8423\n",
      "Epoch 24/35\n",
      "3s - loss: 0.3942 - acc: 0.8458 - val_loss: 0.4087 - val_acc: 0.8429\n",
      "Epoch 25/35\n",
      "2s - loss: 0.3937 - acc: 0.8462 - val_loss: 0.4089 - val_acc: 0.8414\n",
      "Epoch 26/35\n",
      "2s - loss: 0.3925 - acc: 0.8462 - val_loss: 0.4086 - val_acc: 0.8410\n",
      "Epoch 27/35\n",
      "2s - loss: 0.3902 - acc: 0.8480 - val_loss: 0.4072 - val_acc: 0.8419\n",
      "Epoch 28/35\n",
      "2s - loss: 0.3891 - acc: 0.8478 - val_loss: 0.4083 - val_acc: 0.8442\n",
      "Epoch 29/35\n",
      "2s - loss: 0.3869 - acc: 0.8483 - val_loss: 0.4098 - val_acc: 0.8431\n",
      "Epoch 30/35\n",
      "2s - loss: 0.3867 - acc: 0.8482 - val_loss: 0.4108 - val_acc: 0.8438\n",
      "Epoch 31/35\n",
      "2s - loss: 0.3852 - acc: 0.8490 - val_loss: 0.4075 - val_acc: 0.8438\n",
      "Epoch 32/35\n",
      "2s - loss: 0.3835 - acc: 0.8495 - val_loss: 0.4079 - val_acc: 0.8437\n",
      "Epoch 33/35\n",
      "2s - loss: 0.3827 - acc: 0.8491 - val_loss: 0.4110 - val_acc: 0.8433\n",
      "Epoch 34/35\n",
      "2s - loss: 0.3813 - acc: 0.8498 - val_loss: 0.4076 - val_acc: 0.8438\n",
      "Epoch 35/35\n",
      "2s - loss: 0.3803 - acc: 0.8508 - val_loss: 0.4114 - val_acc: 0.8440\n",
      "19648/20000 [============================>.] - ETA: 0sva acc: 0.838125\n",
      "te acc: 0.84405\n",
      "2017-01-12 20:46:11.244749 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4388 - acc: 0.8253 - val_loss: 0.4281 - val_acc: 0.8340\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4276 - acc: 0.8342 - val_loss: 0.4313 - val_acc: 0.8297\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4255 - acc: 0.8351 - val_loss: 0.4260 - val_acc: 0.8329\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4236 - acc: 0.8361 - val_loss: 0.4260 - val_acc: 0.8333\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4219 - acc: 0.8367 - val_loss: 0.4235 - val_acc: 0.8376\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4199 - acc: 0.8367 - val_loss: 0.4225 - val_acc: 0.8364\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4185 - acc: 0.8377 - val_loss: 0.4205 - val_acc: 0.8372\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4163 - acc: 0.8387 - val_loss: 0.4242 - val_acc: 0.8324\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4151 - acc: 0.8387 - val_loss: 0.4179 - val_acc: 0.8399\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4131 - acc: 0.8388 - val_loss: 0.4176 - val_acc: 0.8375\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4125 - acc: 0.8391 - val_loss: 0.4172 - val_acc: 0.8377\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4106 - acc: 0.8406 - val_loss: 0.4172 - val_acc: 0.8358\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4101 - acc: 0.8409 - val_loss: 0.4155 - val_acc: 0.8386\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4086 - acc: 0.8403 - val_loss: 0.4157 - val_acc: 0.8363\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4070 - acc: 0.8413 - val_loss: 0.4133 - val_acc: 0.8405\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4060 - acc: 0.8415 - val_loss: 0.4141 - val_acc: 0.8362\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4038 - acc: 0.8422 - val_loss: 0.4122 - val_acc: 0.8416\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4025 - acc: 0.8425 - val_loss: 0.4116 - val_acc: 0.8407\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4011 - acc: 0.8425 - val_loss: 0.4092 - val_acc: 0.8400\n",
      "Epoch 20/35\n",
      "2s - loss: 0.3997 - acc: 0.8437 - val_loss: 0.4102 - val_acc: 0.8406\n",
      "Epoch 21/35\n",
      "2s - loss: 0.3980 - acc: 0.8441 - val_loss: 0.4102 - val_acc: 0.8399\n",
      "Epoch 22/35\n",
      "2s - loss: 0.3965 - acc: 0.8433 - val_loss: 0.4110 - val_acc: 0.8373\n",
      "Epoch 23/35\n",
      "2s - loss: 0.3952 - acc: 0.8445 - val_loss: 0.4083 - val_acc: 0.8421\n",
      "Epoch 24/35\n",
      "2s - loss: 0.3946 - acc: 0.8451 - val_loss: 0.4072 - val_acc: 0.8427\n",
      "Epoch 25/35\n",
      "2s - loss: 0.3933 - acc: 0.8445 - val_loss: 0.4142 - val_acc: 0.8357\n",
      "Epoch 26/35\n",
      "2s - loss: 0.3920 - acc: 0.8456 - val_loss: 0.4107 - val_acc: 0.8424\n",
      "Epoch 27/35\n",
      "2s - loss: 0.3911 - acc: 0.8465 - val_loss: 0.4097 - val_acc: 0.8434\n",
      "Epoch 28/35\n",
      "2s - loss: 0.3894 - acc: 0.8475 - val_loss: 0.4076 - val_acc: 0.8430\n",
      "Epoch 29/35\n",
      "2s - loss: 0.3875 - acc: 0.8472 - val_loss: 0.4055 - val_acc: 0.8414\n",
      "Epoch 30/35\n",
      "2s - loss: 0.3867 - acc: 0.8480 - val_loss: 0.4071 - val_acc: 0.8429\n",
      "Epoch 31/35\n",
      "2s - loss: 0.3856 - acc: 0.8476 - val_loss: 0.4073 - val_acc: 0.8437\n",
      "Epoch 32/35\n",
      "2s - loss: 0.3832 - acc: 0.8488 - val_loss: 0.4060 - val_acc: 0.8429\n",
      "Epoch 33/35\n",
      "2s - loss: 0.3823 - acc: 0.8491 - val_loss: 0.4081 - val_acc: 0.8438\n",
      "Epoch 34/35\n",
      "2s - loss: 0.3808 - acc: 0.8493 - val_loss: 0.4065 - val_acc: 0.8440\n",
      "Epoch 35/35\n",
      "2s - loss: 0.3796 - acc: 0.8502 - val_loss: 0.4065 - val_acc: 0.8436\n",
      "19872/20000 [============================>.] - ETA: 0sva acc: 0.839\n",
      "te acc: 0.84355\n",
      "2017-01-12 20:47:43.760573 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4411 - acc: 0.8234 - val_loss: 0.4264 - val_acc: 0.8365\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4290 - acc: 0.8341 - val_loss: 0.4294 - val_acc: 0.8308\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4268 - acc: 0.8338 - val_loss: 0.4265 - val_acc: 0.8357\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4251 - acc: 0.8353 - val_loss: 0.4261 - val_acc: 0.8324\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4226 - acc: 0.8363 - val_loss: 0.4229 - val_acc: 0.8371\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4216 - acc: 0.8369 - val_loss: 0.4259 - val_acc: 0.8381\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4197 - acc: 0.8368 - val_loss: 0.4203 - val_acc: 0.8376\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4175 - acc: 0.8373 - val_loss: 0.4233 - val_acc: 0.8377\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4160 - acc: 0.8379 - val_loss: 0.4198 - val_acc: 0.8367\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4148 - acc: 0.8387 - val_loss: 0.4175 - val_acc: 0.8381\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4132 - acc: 0.8389 - val_loss: 0.4171 - val_acc: 0.8381\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4118 - acc: 0.8390 - val_loss: 0.4155 - val_acc: 0.8395\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4103 - acc: 0.8406 - val_loss: 0.4160 - val_acc: 0.8397\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4088 - acc: 0.8405 - val_loss: 0.4152 - val_acc: 0.8384\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4074 - acc: 0.8404 - val_loss: 0.4154 - val_acc: 0.8399\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4067 - acc: 0.8410 - val_loss: 0.4126 - val_acc: 0.8401\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4047 - acc: 0.8415 - val_loss: 0.4121 - val_acc: 0.8406\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4029 - acc: 0.8420 - val_loss: 0.4119 - val_acc: 0.8420\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4022 - acc: 0.8431 - val_loss: 0.4101 - val_acc: 0.8402\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4001 - acc: 0.8433 - val_loss: 0.4122 - val_acc: 0.8392\n",
      "Epoch 21/35\n",
      "2s - loss: 0.3989 - acc: 0.8434 - val_loss: 0.4097 - val_acc: 0.8410\n",
      "Epoch 22/35\n",
      "2s - loss: 0.3972 - acc: 0.8436 - val_loss: 0.4087 - val_acc: 0.8434\n",
      "Epoch 23/35\n",
      "2s - loss: 0.3965 - acc: 0.8450 - val_loss: 0.4077 - val_acc: 0.8422\n",
      "Epoch 24/35\n",
      "2s - loss: 0.3951 - acc: 0.8443 - val_loss: 0.4079 - val_acc: 0.8409\n",
      "Epoch 25/35\n",
      "2s - loss: 0.3938 - acc: 0.8460 - val_loss: 0.4080 - val_acc: 0.8431\n",
      "Epoch 26/35\n",
      "2s - loss: 0.3927 - acc: 0.8457 - val_loss: 0.4101 - val_acc: 0.8402\n",
      "Epoch 27/35\n",
      "2s - loss: 0.3920 - acc: 0.8453 - val_loss: 0.4088 - val_acc: 0.8436\n",
      "Epoch 28/35\n",
      "2s - loss: 0.3896 - acc: 0.8464 - val_loss: 0.4071 - val_acc: 0.8415\n",
      "Epoch 29/35\n",
      "2s - loss: 0.3895 - acc: 0.8464 - val_loss: 0.4081 - val_acc: 0.8425\n",
      "Epoch 30/35\n",
      "2s - loss: 0.3879 - acc: 0.8467 - val_loss: 0.4073 - val_acc: 0.8430\n",
      "Epoch 31/35\n",
      "2s - loss: 0.3874 - acc: 0.8471 - val_loss: 0.4077 - val_acc: 0.8424\n",
      "Epoch 32/35\n",
      "2s - loss: 0.3858 - acc: 0.8485 - val_loss: 0.4075 - val_acc: 0.8418\n",
      "Epoch 33/35\n",
      "2s - loss: 0.3844 - acc: 0.8478 - val_loss: 0.4094 - val_acc: 0.8417\n",
      "Epoch 34/35\n",
      "2s - loss: 0.3829 - acc: 0.8488 - val_loss: 0.4091 - val_acc: 0.8420\n",
      "Epoch 35/35\n",
      "2s - loss: 0.3817 - acc: 0.8493 - val_loss: 0.4085 - val_acc: 0.8427\n",
      "19808/20000 [============================>.] - ETA: 0sva acc: 0.84425\n",
      "te acc: 0.8427\n",
      "2017-01-12 20:49:16.587774 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4407 - acc: 0.8256 - val_loss: 0.4263 - val_acc: 0.8361\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4302 - acc: 0.8326 - val_loss: 0.4270 - val_acc: 0.8360\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4279 - acc: 0.8337 - val_loss: 0.4246 - val_acc: 0.8387\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4263 - acc: 0.8352 - val_loss: 0.4281 - val_acc: 0.8330\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4247 - acc: 0.8346 - val_loss: 0.4239 - val_acc: 0.8380\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4226 - acc: 0.8356 - val_loss: 0.4219 - val_acc: 0.8369\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4211 - acc: 0.8358 - val_loss: 0.4221 - val_acc: 0.8373\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4196 - acc: 0.8372 - val_loss: 0.4210 - val_acc: 0.8353\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4181 - acc: 0.8368 - val_loss: 0.4202 - val_acc: 0.8388\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4171 - acc: 0.8370 - val_loss: 0.4190 - val_acc: 0.8357\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4154 - acc: 0.8383 - val_loss: 0.4217 - val_acc: 0.8327\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4144 - acc: 0.8383 - val_loss: 0.4173 - val_acc: 0.8394\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4130 - acc: 0.8392 - val_loss: 0.4172 - val_acc: 0.8387\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4116 - acc: 0.8387 - val_loss: 0.4170 - val_acc: 0.8408\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4105 - acc: 0.8395 - val_loss: 0.4149 - val_acc: 0.8379\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4092 - acc: 0.8400 - val_loss: 0.4140 - val_acc: 0.8380\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4073 - acc: 0.8405 - val_loss: 0.4135 - val_acc: 0.8386\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4060 - acc: 0.8417 - val_loss: 0.4153 - val_acc: 0.8367\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4045 - acc: 0.8413 - val_loss: 0.4118 - val_acc: 0.8388\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4033 - acc: 0.8420 - val_loss: 0.4121 - val_acc: 0.8413\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4015 - acc: 0.8426 - val_loss: 0.4086 - val_acc: 0.8399\n",
      "Epoch 22/35\n",
      "2s - loss: 0.4005 - acc: 0.8426 - val_loss: 0.4097 - val_acc: 0.8423\n",
      "Epoch 23/35\n",
      "2s - loss: 0.3992 - acc: 0.8430 - val_loss: 0.4086 - val_acc: 0.8413\n",
      "Epoch 24/35\n",
      "2s - loss: 0.3973 - acc: 0.8436 - val_loss: 0.4082 - val_acc: 0.8417\n",
      "Epoch 25/35\n",
      "2s - loss: 0.3962 - acc: 0.8442 - val_loss: 0.4097 - val_acc: 0.8407\n",
      "Epoch 26/35\n",
      "2s - loss: 0.3952 - acc: 0.8444 - val_loss: 0.4075 - val_acc: 0.8415\n",
      "Epoch 27/35\n",
      "2s - loss: 0.3935 - acc: 0.8451 - val_loss: 0.4080 - val_acc: 0.8425\n",
      "Epoch 28/35\n",
      "2s - loss: 0.3934 - acc: 0.8449 - val_loss: 0.4082 - val_acc: 0.8424\n",
      "Epoch 29/35\n",
      "2s - loss: 0.3916 - acc: 0.8454 - val_loss: 0.4109 - val_acc: 0.8396\n",
      "Epoch 30/35\n",
      "2s - loss: 0.3900 - acc: 0.8459 - val_loss: 0.4095 - val_acc: 0.8429\n",
      "Epoch 31/35\n",
      "2s - loss: 0.3894 - acc: 0.8468 - val_loss: 0.4072 - val_acc: 0.8417\n",
      "Epoch 32/35\n",
      "2s - loss: 0.3875 - acc: 0.8471 - val_loss: 0.4086 - val_acc: 0.8427\n",
      "Epoch 33/35\n",
      "2s - loss: 0.3863 - acc: 0.8470 - val_loss: 0.4083 - val_acc: 0.8436\n",
      "Epoch 34/35\n",
      "2s - loss: 0.3859 - acc: 0.8470 - val_loss: 0.4068 - val_acc: 0.8435\n",
      "Epoch 35/35\n",
      "2s - loss: 0.3827 - acc: 0.8487 - val_loss: 0.4086 - val_acc: 0.8436\n",
      "19584/20000 [============================>.] - ETA: 0sva acc: 0.845125\n",
      "te acc: 0.84355\n",
      "2017-01-12 20:50:54.187276 save dbowd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "'''dbow-nn stack for education/age/gender'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "import cfg\n",
    "\n",
    "#-----------------------myfunc-----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "#-----------------------load dataset----------------------\n",
    "df_all = pd.read_csv(cfg.data_path + 'all_v2.csv',encoding='utf8',usecols=['Id','Education','age','gender'],nrows=100000)\n",
    "ys = {}\n",
    "for label in ['Education','age','gender']:\n",
    "    ys[label] = np.array(df_all[label])\n",
    "    \n",
    "model = Doc2Vec.load(cfg.data_path + 'dbow_d2v.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(100000)])\n",
    "\n",
    "#----------------------dbowd2v stack for Education/age/gender---------------------------\n",
    "df_stack = pd.DataFrame(index=range(len(df_all)))\n",
    "TR = 80000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "\n",
    "feat = 'dbowd2v'\n",
    "for i,lb in enumerate(['Education','age','gender']):\n",
    "    num_class = len(pd.value_counts(ys[lb]))\n",
    "    y = ys[lb][:TR]\n",
    "    y_te = ys[lb][TR:]\n",
    "    \n",
    "    stack = np.zeros((X.shape[0],num_class))\n",
    "    stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "    \n",
    "    for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "        print('{} stack:{}/{}'.format(datetime.now(),k+1,n))\n",
    "        nb_classes = num_class\n",
    "        X_train = X[tr]\n",
    "        y_train = y[tr]\n",
    "        X_test = X_te\n",
    "        y_test = y_te\n",
    "\n",
    "        X_train = X_train.astype('float32')\n",
    "        X_test = X_test.astype('float32')\n",
    "        Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "        Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "\n",
    "        model = Sequential()\n",
    "        model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "        model.add(Dropout(0.1))\n",
    "        model.add(Activation('tanh'))\n",
    "        model.add(Dense(nb_classes))\n",
    "        model.add(Activation('softmax'))\n",
    "\n",
    "        model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "        history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=35,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "        y_pred_va = model.predict_proba(X[va])\n",
    "        y_pred_te = model.predict_proba(X_te)\n",
    "        print('va acc:',myAcc(y[va],y_pred_va))\n",
    "        print('te acc:',myAcc(y_te,y_pred_te))\n",
    "        stack[va] += y_pred_va\n",
    "        stack_te += y_pred_te\n",
    "    stack_te /= n\n",
    "    stack_all = np.vstack([stack,stack_te])\n",
    "    for l in range(stack_all.shape[1]):\n",
    "        df_stack['{}_{}_{}'.format(feat,lb,l)] = stack_all[:,l]\n",
    "df_stack.to_csv(cfg.data_path + 'dbowd2v_stack_10W.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dbowd2v stack done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-01-12 20:51:05.276891 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0036 - acc: 0.6026 - val_loss: 0.9368 - val_acc: 0.6299\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9425 - acc: 0.6299 - val_loss: 0.9282 - val_acc: 0.6383\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9373 - acc: 0.6314 - val_loss: 0.9264 - val_acc: 0.6380\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9354 - acc: 0.6316 - val_loss: 0.9254 - val_acc: 0.6412\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9331 - acc: 0.6324 - val_loss: 0.9316 - val_acc: 0.6332\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9315 - acc: 0.6331 - val_loss: 0.9288 - val_acc: 0.6385\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9301 - acc: 0.6337 - val_loss: 0.9243 - val_acc: 0.6410\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9285 - acc: 0.6350 - val_loss: 0.9275 - val_acc: 0.6394\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9267 - acc: 0.6338 - val_loss: 0.9280 - val_acc: 0.6375\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9261 - acc: 0.6345 - val_loss: 0.9236 - val_acc: 0.6421\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9245 - acc: 0.6356 - val_loss: 0.9250 - val_acc: 0.6391\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9243 - acc: 0.6377 - val_loss: 0.9269 - val_acc: 0.6381\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9229 - acc: 0.6365 - val_loss: 0.9243 - val_acc: 0.6390\n",
      "Epoch 14/35\n",
      "2s - loss: 0.9219 - acc: 0.6368 - val_loss: 0.9235 - val_acc: 0.6408\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9211 - acc: 0.6375 - val_loss: 0.9273 - val_acc: 0.6365\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9188 - acc: 0.6379 - val_loss: 0.9222 - val_acc: 0.6409\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9182 - acc: 0.6394 - val_loss: 0.9253 - val_acc: 0.6398\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9165 - acc: 0.6383 - val_loss: 0.9270 - val_acc: 0.6356\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9149 - acc: 0.6381 - val_loss: 0.9210 - val_acc: 0.6414\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9136 - acc: 0.6402 - val_loss: 0.9213 - val_acc: 0.6394\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9118 - acc: 0.6405 - val_loss: 0.9183 - val_acc: 0.6424\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9097 - acc: 0.6418 - val_loss: 0.9187 - val_acc: 0.6431\n",
      "Epoch 23/35\n",
      "3s - loss: 0.9077 - acc: 0.6424 - val_loss: 0.9209 - val_acc: 0.6409\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9066 - acc: 0.6417 - val_loss: 0.9164 - val_acc: 0.6444\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9050 - acc: 0.6445 - val_loss: 0.9147 - val_acc: 0.6446\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9030 - acc: 0.6452 - val_loss: 0.9153 - val_acc: 0.6432\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9007 - acc: 0.6460 - val_loss: 0.9132 - val_acc: 0.6451\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8996 - acc: 0.6461 - val_loss: 0.9123 - val_acc: 0.6464\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8972 - acc: 0.6472 - val_loss: 0.9118 - val_acc: 0.6468\n",
      "Epoch 30/35\n",
      "3s - loss: 0.8957 - acc: 0.6472 - val_loss: 0.9108 - val_acc: 0.6465\n",
      "Epoch 31/35\n",
      "3s - loss: 0.8940 - acc: 0.6475 - val_loss: 0.9124 - val_acc: 0.6455\n",
      "Epoch 32/35\n",
      "3s - loss: 0.8922 - acc: 0.6501 - val_loss: 0.9117 - val_acc: 0.6467\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8912 - acc: 0.6497 - val_loss: 0.9109 - val_acc: 0.6456\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8889 - acc: 0.6517 - val_loss: 0.9117 - val_acc: 0.6445\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8875 - acc: 0.6517 - val_loss: 0.9086 - val_acc: 0.6480\n",
      "19168/20000 [===========================>..] - ETA: 0sva acc: 0.64225\n",
      "te acc: 0.64795\n",
      "2017-01-12 20:52:45.486203 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0015 - acc: 0.6016 - val_loss: 0.9415 - val_acc: 0.6303\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9384 - acc: 0.6308 - val_loss: 0.9333 - val_acc: 0.6322\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9335 - acc: 0.6314 - val_loss: 0.9284 - val_acc: 0.6355\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9317 - acc: 0.6320 - val_loss: 0.9271 - val_acc: 0.6359\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9298 - acc: 0.6333 - val_loss: 0.9276 - val_acc: 0.6371\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9281 - acc: 0.6324 - val_loss: 0.9271 - val_acc: 0.6395\n",
      "Epoch 7/35\n",
      "3s - loss: 0.9256 - acc: 0.6340 - val_loss: 0.9260 - val_acc: 0.6401\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9256 - acc: 0.6362 - val_loss: 0.9257 - val_acc: 0.6394\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9238 - acc: 0.6357 - val_loss: 0.9253 - val_acc: 0.6394\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9225 - acc: 0.6355 - val_loss: 0.9290 - val_acc: 0.6372\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9216 - acc: 0.6375 - val_loss: 0.9278 - val_acc: 0.6387\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9207 - acc: 0.6355 - val_loss: 0.9263 - val_acc: 0.6411\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9196 - acc: 0.6370 - val_loss: 0.9279 - val_acc: 0.6389\n",
      "Epoch 14/35\n",
      "2s - loss: 0.9182 - acc: 0.6382 - val_loss: 0.9276 - val_acc: 0.6377\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9174 - acc: 0.6375 - val_loss: 0.9250 - val_acc: 0.6405\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9162 - acc: 0.6384 - val_loss: 0.9238 - val_acc: 0.6403\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9151 - acc: 0.6395 - val_loss: 0.9274 - val_acc: 0.6391\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9134 - acc: 0.6396 - val_loss: 0.9237 - val_acc: 0.6404\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9122 - acc: 0.6409 - val_loss: 0.9240 - val_acc: 0.6415\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9105 - acc: 0.6395 - val_loss: 0.9231 - val_acc: 0.6415\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9095 - acc: 0.6404 - val_loss: 0.9210 - val_acc: 0.6420\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9067 - acc: 0.6417 - val_loss: 0.9196 - val_acc: 0.6443\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9048 - acc: 0.6423 - val_loss: 0.9191 - val_acc: 0.6421\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9045 - acc: 0.6415 - val_loss: 0.9193 - val_acc: 0.6420\n",
      "Epoch 25/35\n",
      "3s - loss: 0.9029 - acc: 0.6437 - val_loss: 0.9205 - val_acc: 0.6419\n",
      "Epoch 26/35\n",
      "3s - loss: 0.9004 - acc: 0.6440 - val_loss: 0.9168 - val_acc: 0.6448\n",
      "Epoch 27/35\n",
      "3s - loss: 0.8991 - acc: 0.6448 - val_loss: 0.9213 - val_acc: 0.6397\n",
      "Epoch 28/35\n",
      "3s - loss: 0.8978 - acc: 0.6457 - val_loss: 0.9163 - val_acc: 0.6438\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8964 - acc: 0.6450 - val_loss: 0.9167 - val_acc: 0.6431\n",
      "Epoch 30/35\n",
      "3s - loss: 0.8940 - acc: 0.6462 - val_loss: 0.9137 - val_acc: 0.6452\n",
      "Epoch 31/35\n",
      "3s - loss: 0.8928 - acc: 0.6472 - val_loss: 0.9176 - val_acc: 0.6435\n",
      "Epoch 32/35\n",
      "3s - loss: 0.8912 - acc: 0.6475 - val_loss: 0.9137 - val_acc: 0.6449\n",
      "Epoch 33/35\n",
      "3s - loss: 0.8893 - acc: 0.6489 - val_loss: 0.9144 - val_acc: 0.6445\n",
      "Epoch 34/35\n",
      "3s - loss: 0.8871 - acc: 0.6493 - val_loss: 0.9118 - val_acc: 0.6453\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8845 - acc: 0.6511 - val_loss: 0.9144 - val_acc: 0.6448\n",
      "19936/20000 [============================>.] - ETA: 0sva acc: 0.641625\n",
      "te acc: 0.64475\n",
      "2017-01-12 20:54:27.454439 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0012 - acc: 0.6060 - val_loss: 0.9319 - val_acc: 0.6368\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9405 - acc: 0.6305 - val_loss: 0.9298 - val_acc: 0.6352\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9357 - acc: 0.6321 - val_loss: 0.9255 - val_acc: 0.6382\n",
      "Epoch 4/35\n",
      "3s - loss: 0.9326 - acc: 0.6332 - val_loss: 0.9259 - val_acc: 0.6381\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9305 - acc: 0.6349 - val_loss: 0.9279 - val_acc: 0.6371\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9288 - acc: 0.6343 - val_loss: 0.9266 - val_acc: 0.6369\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9271 - acc: 0.6335 - val_loss: 0.9253 - val_acc: 0.6407\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9259 - acc: 0.6358 - val_loss: 0.9261 - val_acc: 0.6401\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9249 - acc: 0.6354 - val_loss: 0.9267 - val_acc: 0.6392\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9238 - acc: 0.6365 - val_loss: 0.9253 - val_acc: 0.6403\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9233 - acc: 0.6356 - val_loss: 0.9311 - val_acc: 0.6364\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9212 - acc: 0.6383 - val_loss: 0.9255 - val_acc: 0.6404\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9203 - acc: 0.6379 - val_loss: 0.9246 - val_acc: 0.6419\n",
      "Epoch 14/35\n",
      "2s - loss: 0.9192 - acc: 0.6378 - val_loss: 0.9248 - val_acc: 0.6403\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9178 - acc: 0.6378 - val_loss: 0.9233 - val_acc: 0.6408\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9163 - acc: 0.6388 - val_loss: 0.9228 - val_acc: 0.6424\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9155 - acc: 0.6387 - val_loss: 0.9240 - val_acc: 0.6407\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9136 - acc: 0.6387 - val_loss: 0.9229 - val_acc: 0.6405\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9123 - acc: 0.6401 - val_loss: 0.9219 - val_acc: 0.6420\n",
      "Epoch 20/35\n",
      "3s - loss: 0.9110 - acc: 0.6399 - val_loss: 0.9214 - val_acc: 0.6405\n",
      "Epoch 21/35\n",
      "3s - loss: 0.9090 - acc: 0.6416 - val_loss: 0.9199 - val_acc: 0.6423\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9073 - acc: 0.6416 - val_loss: 0.9187 - val_acc: 0.6438\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9058 - acc: 0.6411 - val_loss: 0.9201 - val_acc: 0.6433\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9047 - acc: 0.6426 - val_loss: 0.9166 - val_acc: 0.6442\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9034 - acc: 0.6434 - val_loss: 0.9175 - val_acc: 0.6427\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9010 - acc: 0.6456 - val_loss: 0.9168 - val_acc: 0.6442\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9004 - acc: 0.6456 - val_loss: 0.9154 - val_acc: 0.6452\n",
      "Epoch 28/35\n",
      "2s - loss: 0.8986 - acc: 0.6454 - val_loss: 0.9162 - val_acc: 0.6446\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8964 - acc: 0.6452 - val_loss: 0.9144 - val_acc: 0.6445\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8959 - acc: 0.6480 - val_loss: 0.9160 - val_acc: 0.6443\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8937 - acc: 0.6478 - val_loss: 0.9125 - val_acc: 0.6464\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8918 - acc: 0.6484 - val_loss: 0.9137 - val_acc: 0.6472\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8911 - acc: 0.6479 - val_loss: 0.9126 - val_acc: 0.6451\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8890 - acc: 0.6500 - val_loss: 0.9133 - val_acc: 0.6474\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8881 - acc: 0.6503 - val_loss: 0.9114 - val_acc: 0.6456\n",
      "19872/20000 [============================>.] - ETA: 0sva acc: 0.634875\n",
      "te acc: 0.6456\n",
      "2017-01-12 20:56:06.034148 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0102 - acc: 0.5995 - val_loss: 0.9313 - val_acc: 0.6383\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9421 - acc: 0.6296 - val_loss: 0.9282 - val_acc: 0.6363\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9372 - acc: 0.6306 - val_loss: 0.9366 - val_acc: 0.6321\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9345 - acc: 0.6297 - val_loss: 0.9342 - val_acc: 0.6316\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9322 - acc: 0.6324 - val_loss: 0.9261 - val_acc: 0.6391\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9309 - acc: 0.6334 - val_loss: 0.9274 - val_acc: 0.6387\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9288 - acc: 0.6328 - val_loss: 0.9257 - val_acc: 0.6400\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9279 - acc: 0.6336 - val_loss: 0.9254 - val_acc: 0.6401\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9262 - acc: 0.6349 - val_loss: 0.9277 - val_acc: 0.6380\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9256 - acc: 0.6342 - val_loss: 0.9253 - val_acc: 0.6403\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9243 - acc: 0.6345 - val_loss: 0.9271 - val_acc: 0.6408\n",
      "Epoch 12/35\n",
      "2s - loss: 0.9235 - acc: 0.6358 - val_loss: 0.9274 - val_acc: 0.6389\n",
      "Epoch 13/35\n",
      "2s - loss: 0.9218 - acc: 0.6371 - val_loss: 0.9266 - val_acc: 0.6412\n",
      "Epoch 14/35\n",
      "2s - loss: 0.9208 - acc: 0.6372 - val_loss: 0.9261 - val_acc: 0.6393\n",
      "Epoch 15/35\n",
      "2s - loss: 0.9191 - acc: 0.6368 - val_loss: 0.9242 - val_acc: 0.6410\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9181 - acc: 0.6374 - val_loss: 0.9267 - val_acc: 0.6405\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9169 - acc: 0.6389 - val_loss: 0.9253 - val_acc: 0.6394\n",
      "Epoch 18/35\n",
      "2s - loss: 0.9149 - acc: 0.6393 - val_loss: 0.9248 - val_acc: 0.6410\n",
      "Epoch 19/35\n",
      "2s - loss: 0.9129 - acc: 0.6398 - val_loss: 0.9225 - val_acc: 0.6400\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9129 - acc: 0.6388 - val_loss: 0.9212 - val_acc: 0.6424\n",
      "Epoch 21/35\n",
      "2s - loss: 0.9103 - acc: 0.6390 - val_loss: 0.9218 - val_acc: 0.6401\n",
      "Epoch 22/35\n",
      "2s - loss: 0.9093 - acc: 0.6410 - val_loss: 0.9190 - val_acc: 0.6421\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9071 - acc: 0.6409 - val_loss: 0.9194 - val_acc: 0.6413\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9057 - acc: 0.6427 - val_loss: 0.9190 - val_acc: 0.6428\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9038 - acc: 0.6437 - val_loss: 0.9176 - val_acc: 0.6432\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9029 - acc: 0.6425 - val_loss: 0.9169 - val_acc: 0.6440\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9004 - acc: 0.6449 - val_loss: 0.9169 - val_acc: 0.6436\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9001 - acc: 0.6445 - val_loss: 0.9167 - val_acc: 0.6432\n",
      "Epoch 29/35\n",
      "2s - loss: 0.8985 - acc: 0.6454 - val_loss: 0.9161 - val_acc: 0.6442\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8965 - acc: 0.6469 - val_loss: 0.9140 - val_acc: 0.6456\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8941 - acc: 0.6459 - val_loss: 0.9146 - val_acc: 0.6457\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8922 - acc: 0.6495 - val_loss: 0.9150 - val_acc: 0.6453\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8902 - acc: 0.6501 - val_loss: 0.9118 - val_acc: 0.6475\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8885 - acc: 0.6506 - val_loss: 0.9145 - val_acc: 0.6453\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8869 - acc: 0.6498 - val_loss: 0.9115 - val_acc: 0.6491\n",
      "19936/20000 [============================>.] - ETA: 0sva acc: 0.645125\n",
      "te acc: 0.64905\n",
      "2017-01-12 20:57:43.177141 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.0076 - acc: 0.6008 - val_loss: 0.9304 - val_acc: 0.6372\n",
      "Epoch 2/35\n",
      "2s - loss: 0.9422 - acc: 0.6282 - val_loss: 0.9271 - val_acc: 0.6358\n",
      "Epoch 3/35\n",
      "2s - loss: 0.9373 - acc: 0.6311 - val_loss: 0.9275 - val_acc: 0.6348\n",
      "Epoch 4/35\n",
      "2s - loss: 0.9349 - acc: 0.6305 - val_loss: 0.9247 - val_acc: 0.6399\n",
      "Epoch 5/35\n",
      "2s - loss: 0.9332 - acc: 0.6315 - val_loss: 0.9255 - val_acc: 0.6396\n",
      "Epoch 6/35\n",
      "2s - loss: 0.9311 - acc: 0.6322 - val_loss: 0.9252 - val_acc: 0.6400\n",
      "Epoch 7/35\n",
      "2s - loss: 0.9295 - acc: 0.6333 - val_loss: 0.9283 - val_acc: 0.6375\n",
      "Epoch 8/35\n",
      "2s - loss: 0.9287 - acc: 0.6338 - val_loss: 0.9257 - val_acc: 0.6400\n",
      "Epoch 9/35\n",
      "2s - loss: 0.9273 - acc: 0.6353 - val_loss: 0.9275 - val_acc: 0.6383\n",
      "Epoch 10/35\n",
      "2s - loss: 0.9267 - acc: 0.6359 - val_loss: 0.9249 - val_acc: 0.6397\n",
      "Epoch 11/35\n",
      "2s - loss: 0.9253 - acc: 0.6343 - val_loss: 0.9254 - val_acc: 0.6401\n",
      "Epoch 12/35\n",
      "3s - loss: 0.9242 - acc: 0.6360 - val_loss: 0.9258 - val_acc: 0.6392\n",
      "Epoch 13/35\n",
      "3s - loss: 0.9229 - acc: 0.6363 - val_loss: 0.9278 - val_acc: 0.6393\n",
      "Epoch 14/35\n",
      "3s - loss: 0.9222 - acc: 0.6362 - val_loss: 0.9247 - val_acc: 0.6421\n",
      "Epoch 15/35\n",
      "3s - loss: 0.9213 - acc: 0.6361 - val_loss: 0.9241 - val_acc: 0.6404\n",
      "Epoch 16/35\n",
      "2s - loss: 0.9201 - acc: 0.6374 - val_loss: 0.9239 - val_acc: 0.6413\n",
      "Epoch 17/35\n",
      "2s - loss: 0.9181 - acc: 0.6382 - val_loss: 0.9233 - val_acc: 0.6428\n",
      "Epoch 18/35\n",
      "3s - loss: 0.9177 - acc: 0.6385 - val_loss: 0.9261 - val_acc: 0.6391\n",
      "Epoch 19/35\n",
      "3s - loss: 0.9164 - acc: 0.6381 - val_loss: 0.9215 - val_acc: 0.6423\n",
      "Epoch 20/35\n",
      "2s - loss: 0.9147 - acc: 0.6385 - val_loss: 0.9218 - val_acc: 0.6421\n",
      "Epoch 21/35\n",
      "3s - loss: 0.9128 - acc: 0.6404 - val_loss: 0.9195 - val_acc: 0.6424\n",
      "Epoch 22/35\n",
      "3s - loss: 0.9116 - acc: 0.6399 - val_loss: 0.9209 - val_acc: 0.6408\n",
      "Epoch 23/35\n",
      "2s - loss: 0.9108 - acc: 0.6404 - val_loss: 0.9199 - val_acc: 0.6401\n",
      "Epoch 24/35\n",
      "2s - loss: 0.9086 - acc: 0.6421 - val_loss: 0.9197 - val_acc: 0.6428\n",
      "Epoch 25/35\n",
      "2s - loss: 0.9074 - acc: 0.6413 - val_loss: 0.9190 - val_acc: 0.6399\n",
      "Epoch 26/35\n",
      "2s - loss: 0.9056 - acc: 0.6423 - val_loss: 0.9162 - val_acc: 0.6437\n",
      "Epoch 27/35\n",
      "2s - loss: 0.9030 - acc: 0.6431 - val_loss: 0.9184 - val_acc: 0.6425\n",
      "Epoch 28/35\n",
      "2s - loss: 0.9021 - acc: 0.6447 - val_loss: 0.9182 - val_acc: 0.6415\n",
      "Epoch 29/35\n",
      "2s - loss: 0.9000 - acc: 0.6450 - val_loss: 0.9136 - val_acc: 0.6453\n",
      "Epoch 30/35\n",
      "2s - loss: 0.8979 - acc: 0.6459 - val_loss: 0.9124 - val_acc: 0.6447\n",
      "Epoch 31/35\n",
      "2s - loss: 0.8963 - acc: 0.6455 - val_loss: 0.9126 - val_acc: 0.6443\n",
      "Epoch 32/35\n",
      "2s - loss: 0.8939 - acc: 0.6476 - val_loss: 0.9134 - val_acc: 0.6448\n",
      "Epoch 33/35\n",
      "2s - loss: 0.8927 - acc: 0.6477 - val_loss: 0.9122 - val_acc: 0.6464\n",
      "Epoch 34/35\n",
      "2s - loss: 0.8913 - acc: 0.6488 - val_loss: 0.9094 - val_acc: 0.6468\n",
      "Epoch 35/35\n",
      "2s - loss: 0.8892 - acc: 0.6503 - val_loss: 0.9131 - val_acc: 0.6476\n",
      "19712/20000 [============================>.] - ETA: 0sva acc: 0.6404375\n",
      "te acc: 0.6476\n",
      "2017-01-12 20:59:24.232831 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1264 - acc: 0.5534 - val_loss: 1.0438 - val_acc: 0.5911\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0554 - acc: 0.5871 - val_loss: 1.0469 - val_acc: 0.5935\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0507 - acc: 0.5895 - val_loss: 1.0429 - val_acc: 0.5959\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0481 - acc: 0.5893 - val_loss: 1.0410 - val_acc: 0.5981\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0461 - acc: 0.5913 - val_loss: 1.0449 - val_acc: 0.5874\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0447 - acc: 0.5914 - val_loss: 1.0393 - val_acc: 0.5940\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0435 - acc: 0.5909 - val_loss: 1.0409 - val_acc: 0.5966\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0417 - acc: 0.5934 - val_loss: 1.0380 - val_acc: 0.5970\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0406 - acc: 0.5932 - val_loss: 1.0373 - val_acc: 0.5969\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0397 - acc: 0.5936 - val_loss: 1.0387 - val_acc: 0.5955\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0375 - acc: 0.5932 - val_loss: 1.0408 - val_acc: 0.5985\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0362 - acc: 0.5947 - val_loss: 1.0374 - val_acc: 0.5947\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0348 - acc: 0.5950 - val_loss: 1.0361 - val_acc: 0.5970\n",
      "Epoch 14/35\n",
      "3s - loss: 1.0334 - acc: 0.5971 - val_loss: 1.0336 - val_acc: 0.5979\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0321 - acc: 0.5962 - val_loss: 1.0374 - val_acc: 0.5981\n",
      "Epoch 16/35\n",
      "2s - loss: 1.0301 - acc: 0.5979 - val_loss: 1.0358 - val_acc: 0.5986\n",
      "Epoch 17/35\n",
      "2s - loss: 1.0276 - acc: 0.5974 - val_loss: 1.0304 - val_acc: 0.5992\n",
      "Epoch 18/35\n",
      "2s - loss: 1.0254 - acc: 0.5989 - val_loss: 1.0353 - val_acc: 0.5997\n",
      "Epoch 19/35\n",
      "2s - loss: 1.0236 - acc: 0.5988 - val_loss: 1.0286 - val_acc: 0.5984\n",
      "Epoch 20/35\n",
      "2s - loss: 1.0209 - acc: 0.6018 - val_loss: 1.0311 - val_acc: 0.5997\n",
      "Epoch 21/35\n",
      "2s - loss: 1.0205 - acc: 0.6006 - val_loss: 1.0271 - val_acc: 0.6034\n",
      "Epoch 22/35\n",
      "2s - loss: 1.0177 - acc: 0.6021 - val_loss: 1.0267 - val_acc: 0.6004\n",
      "Epoch 23/35\n",
      "2s - loss: 1.0154 - acc: 0.6016 - val_loss: 1.0241 - val_acc: 0.6010\n",
      "Epoch 24/35\n",
      "2s - loss: 1.0137 - acc: 0.6027 - val_loss: 1.0232 - val_acc: 0.6030\n",
      "Epoch 25/35\n",
      "2s - loss: 1.0124 - acc: 0.6024 - val_loss: 1.0220 - val_acc: 0.6020\n",
      "Epoch 26/35\n",
      "2s - loss: 1.0100 - acc: 0.6036 - val_loss: 1.0219 - val_acc: 0.6028\n",
      "Epoch 27/35\n",
      "2s - loss: 1.0091 - acc: 0.6037 - val_loss: 1.0218 - val_acc: 0.6024\n",
      "Epoch 28/35\n",
      "2s - loss: 1.0059 - acc: 0.6063 - val_loss: 1.0219 - val_acc: 0.6036\n",
      "Epoch 29/35\n",
      "2s - loss: 1.0050 - acc: 0.6062 - val_loss: 1.0200 - val_acc: 0.6022\n",
      "Epoch 30/35\n",
      "2s - loss: 1.0025 - acc: 0.6072 - val_loss: 1.0203 - val_acc: 0.6053\n",
      "Epoch 31/35\n",
      "2s - loss: 1.0008 - acc: 0.6084 - val_loss: 1.0202 - val_acc: 0.6024\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9988 - acc: 0.6082 - val_loss: 1.0183 - val_acc: 0.6020\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9984 - acc: 0.6107 - val_loss: 1.0224 - val_acc: 0.6030\n",
      "Epoch 34/35\n",
      "2s - loss: 0.9959 - acc: 0.6089 - val_loss: 1.0242 - val_acc: 0.6048\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9945 - acc: 0.6116 - val_loss: 1.0168 - val_acc: 0.6048\n",
      "19936/20000 [============================>.] - ETA: 0sva acc: 0.601875\n",
      "te acc: 0.6048\n",
      "2017-01-12 21:01:01.394445 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1260 - acc: 0.5516 - val_loss: 1.0426 - val_acc: 0.5948\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0526 - acc: 0.5874 - val_loss: 1.0395 - val_acc: 0.5945\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0488 - acc: 0.5871 - val_loss: 1.0439 - val_acc: 0.5952\n",
      "Epoch 4/35\n",
      "3s - loss: 1.0461 - acc: 0.5904 - val_loss: 1.0403 - val_acc: 0.5904\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0434 - acc: 0.5896 - val_loss: 1.0435 - val_acc: 0.5967\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0416 - acc: 0.5918 - val_loss: 1.0402 - val_acc: 0.5937\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0406 - acc: 0.5930 - val_loss: 1.0437 - val_acc: 0.5976\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0393 - acc: 0.5913 - val_loss: 1.0391 - val_acc: 0.5948\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0382 - acc: 0.5938 - val_loss: 1.0411 - val_acc: 0.5960\n",
      "Epoch 10/35\n",
      "3s - loss: 1.0361 - acc: 0.5943 - val_loss: 1.0385 - val_acc: 0.5963\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0349 - acc: 0.5942 - val_loss: 1.0384 - val_acc: 0.5980\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0338 - acc: 0.5950 - val_loss: 1.0372 - val_acc: 0.5979\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0324 - acc: 0.5948 - val_loss: 1.0427 - val_acc: 0.5987\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0311 - acc: 0.5965 - val_loss: 1.0380 - val_acc: 0.5943\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0287 - acc: 0.5966 - val_loss: 1.0367 - val_acc: 0.5951\n",
      "Epoch 16/35\n",
      "2s - loss: 1.0273 - acc: 0.5967 - val_loss: 1.0331 - val_acc: 0.5968\n",
      "Epoch 17/35\n",
      "2s - loss: 1.0243 - acc: 0.5979 - val_loss: 1.0328 - val_acc: 0.5990\n",
      "Epoch 18/35\n",
      "2s - loss: 1.0228 - acc: 0.5987 - val_loss: 1.0330 - val_acc: 0.5982\n",
      "Epoch 19/35\n",
      "2s - loss: 1.0203 - acc: 0.5986 - val_loss: 1.0297 - val_acc: 0.5986\n",
      "Epoch 20/35\n",
      "2s - loss: 1.0181 - acc: 0.6012 - val_loss: 1.0297 - val_acc: 0.5981\n",
      "Epoch 21/35\n",
      "2s - loss: 1.0164 - acc: 0.6018 - val_loss: 1.0283 - val_acc: 0.6018\n",
      "Epoch 22/35\n",
      "2s - loss: 1.0140 - acc: 0.6015 - val_loss: 1.0283 - val_acc: 0.6027\n",
      "Epoch 23/35\n",
      "2s - loss: 1.0126 - acc: 0.6006 - val_loss: 1.0249 - val_acc: 0.6019\n",
      "Epoch 24/35\n",
      "2s - loss: 1.0103 - acc: 0.6042 - val_loss: 1.0237 - val_acc: 0.6004\n",
      "Epoch 25/35\n",
      "3s - loss: 1.0088 - acc: 0.6042 - val_loss: 1.0222 - val_acc: 0.6001\n",
      "Epoch 26/35\n",
      "3s - loss: 1.0062 - acc: 0.6041 - val_loss: 1.0282 - val_acc: 0.6026\n",
      "Epoch 27/35\n",
      "2s - loss: 1.0050 - acc: 0.6047 - val_loss: 1.0235 - val_acc: 0.6034\n",
      "Epoch 28/35\n",
      "3s - loss: 1.0031 - acc: 0.6061 - val_loss: 1.0195 - val_acc: 0.6048\n",
      "Epoch 29/35\n",
      "2s - loss: 1.0012 - acc: 0.6073 - val_loss: 1.0277 - val_acc: 0.6017\n",
      "Epoch 30/35\n",
      "2s - loss: 0.9995 - acc: 0.6080 - val_loss: 1.0204 - val_acc: 0.6052\n",
      "Epoch 31/35\n",
      "2s - loss: 0.9987 - acc: 0.6075 - val_loss: 1.0181 - val_acc: 0.6058\n",
      "Epoch 32/35\n",
      "3s - loss: 0.9967 - acc: 0.6077 - val_loss: 1.0207 - val_acc: 0.6047\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9956 - acc: 0.6086 - val_loss: 1.0171 - val_acc: 0.6048\n",
      "Epoch 34/35\n",
      "3s - loss: 0.9929 - acc: 0.6096 - val_loss: 1.0189 - val_acc: 0.6055\n",
      "Epoch 35/35\n",
      "3s - loss: 0.9911 - acc: 0.6119 - val_loss: 1.0182 - val_acc: 0.6056\n",
      "20000/20000 [==============================] - 0s     \n",
      "va acc: 0.598125\n",
      "te acc: 0.60555\n",
      "2017-01-12 21:02:44.096061 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "3s - loss: 1.1247 - acc: 0.5563 - val_loss: 1.0449 - val_acc: 0.5896\n",
      "Epoch 2/35\n",
      "3s - loss: 1.0556 - acc: 0.5870 - val_loss: 1.0403 - val_acc: 0.5948\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0514 - acc: 0.5889 - val_loss: 1.0383 - val_acc: 0.5911\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0495 - acc: 0.5895 - val_loss: 1.0390 - val_acc: 0.5941\n",
      "Epoch 5/35\n",
      "3s - loss: 1.0471 - acc: 0.5908 - val_loss: 1.0377 - val_acc: 0.5945\n",
      "Epoch 6/35\n",
      "3s - loss: 1.0453 - acc: 0.5901 - val_loss: 1.0392 - val_acc: 0.5966\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0441 - acc: 0.5910 - val_loss: 1.0412 - val_acc: 0.5970\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0423 - acc: 0.5913 - val_loss: 1.0362 - val_acc: 0.5978\n",
      "Epoch 9/35\n",
      "3s - loss: 1.0416 - acc: 0.5929 - val_loss: 1.0363 - val_acc: 0.5967\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0406 - acc: 0.5936 - val_loss: 1.0363 - val_acc: 0.5990\n",
      "Epoch 11/35\n",
      "3s - loss: 1.0387 - acc: 0.5932 - val_loss: 1.0378 - val_acc: 0.5992\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0373 - acc: 0.5938 - val_loss: 1.0378 - val_acc: 0.5982\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0357 - acc: 0.5945 - val_loss: 1.0340 - val_acc: 0.5987\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0331 - acc: 0.5967 - val_loss: 1.0360 - val_acc: 0.5996\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0315 - acc: 0.5975 - val_loss: 1.0334 - val_acc: 0.5982\n",
      "Epoch 16/35\n",
      "2s - loss: 1.0294 - acc: 0.5959 - val_loss: 1.0335 - val_acc: 0.5957\n",
      "Epoch 17/35\n",
      "2s - loss: 1.0268 - acc: 0.5995 - val_loss: 1.0291 - val_acc: 0.5978\n",
      "Epoch 18/35\n",
      "2s - loss: 1.0251 - acc: 0.5985 - val_loss: 1.0325 - val_acc: 0.6005\n",
      "Epoch 19/35\n",
      "2s - loss: 1.0228 - acc: 0.5990 - val_loss: 1.0262 - val_acc: 0.6031\n",
      "Epoch 20/35\n",
      "2s - loss: 1.0201 - acc: 0.6015 - val_loss: 1.0327 - val_acc: 0.6002\n",
      "Epoch 21/35\n",
      "2s - loss: 1.0174 - acc: 0.6017 - val_loss: 1.0277 - val_acc: 0.5993\n",
      "Epoch 22/35\n",
      "2s - loss: 1.0165 - acc: 0.6018 - val_loss: 1.0231 - val_acc: 0.6038\n",
      "Epoch 23/35\n",
      "3s - loss: 1.0135 - acc: 0.6058 - val_loss: 1.0310 - val_acc: 0.6002\n",
      "Epoch 24/35\n",
      "2s - loss: 1.0124 - acc: 0.6051 - val_loss: 1.0203 - val_acc: 0.6031\n",
      "Epoch 25/35\n",
      "2s - loss: 1.0103 - acc: 0.6066 - val_loss: 1.0191 - val_acc: 0.6038\n",
      "Epoch 26/35\n",
      "2s - loss: 1.0086 - acc: 0.6046 - val_loss: 1.0246 - val_acc: 0.6028\n",
      "Epoch 27/35\n",
      "2s - loss: 1.0054 - acc: 0.6058 - val_loss: 1.0182 - val_acc: 0.6037\n",
      "Epoch 28/35\n",
      "2s - loss: 1.0041 - acc: 0.6079 - val_loss: 1.0160 - val_acc: 0.6031\n",
      "Epoch 29/35\n",
      "2s - loss: 1.0028 - acc: 0.6069 - val_loss: 1.0147 - val_acc: 0.6032\n",
      "Epoch 30/35\n",
      "2s - loss: 1.0008 - acc: 0.6076 - val_loss: 1.0176 - val_acc: 0.6055\n",
      "Epoch 31/35\n",
      "2s - loss: 0.9981 - acc: 0.6093 - val_loss: 1.0156 - val_acc: 0.6045\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9972 - acc: 0.6096 - val_loss: 1.0144 - val_acc: 0.6038\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9961 - acc: 0.6102 - val_loss: 1.0138 - val_acc: 0.6057\n",
      "Epoch 34/35\n",
      "2s - loss: 0.9931 - acc: 0.6129 - val_loss: 1.0140 - val_acc: 0.6053\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9912 - acc: 0.6138 - val_loss: 1.0168 - val_acc: 0.6047\n",
      "19840/20000 [============================>.] - ETA: 0sva acc: 0.5966875\n",
      "te acc: 0.60465\n",
      "2017-01-12 21:04:24.926676 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1197 - acc: 0.5586 - val_loss: 1.0467 - val_acc: 0.5920\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0561 - acc: 0.5868 - val_loss: 1.0435 - val_acc: 0.5933\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0515 - acc: 0.5878 - val_loss: 1.0393 - val_acc: 0.5969\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0485 - acc: 0.5895 - val_loss: 1.0402 - val_acc: 0.5937\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0457 - acc: 0.5900 - val_loss: 1.0423 - val_acc: 0.5955\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0452 - acc: 0.5910 - val_loss: 1.0422 - val_acc: 0.5914\n",
      "Epoch 7/35\n",
      "2s - loss: 1.0434 - acc: 0.5916 - val_loss: 1.0402 - val_acc: 0.5929\n",
      "Epoch 8/35\n",
      "2s - loss: 1.0420 - acc: 0.5932 - val_loss: 1.0412 - val_acc: 0.5970\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0401 - acc: 0.5927 - val_loss: 1.0420 - val_acc: 0.5939\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0394 - acc: 0.5950 - val_loss: 1.0411 - val_acc: 0.5970\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0382 - acc: 0.5937 - val_loss: 1.0372 - val_acc: 0.5944\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0373 - acc: 0.5939 - val_loss: 1.0368 - val_acc: 0.5967\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0352 - acc: 0.5941 - val_loss: 1.0366 - val_acc: 0.5988\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0344 - acc: 0.5950 - val_loss: 1.0369 - val_acc: 0.5967\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0333 - acc: 0.5956 - val_loss: 1.0348 - val_acc: 0.5975\n",
      "Epoch 16/35\n",
      "2s - loss: 1.0304 - acc: 0.5964 - val_loss: 1.0349 - val_acc: 0.5988\n",
      "Epoch 17/35\n",
      "2s - loss: 1.0284 - acc: 0.5982 - val_loss: 1.0344 - val_acc: 0.5959\n",
      "Epoch 18/35\n",
      "2s - loss: 1.0264 - acc: 0.5977 - val_loss: 1.0315 - val_acc: 0.5977\n",
      "Epoch 19/35\n",
      "2s - loss: 1.0249 - acc: 0.5997 - val_loss: 1.0325 - val_acc: 0.5991\n",
      "Epoch 20/35\n",
      "2s - loss: 1.0225 - acc: 0.6005 - val_loss: 1.0338 - val_acc: 0.5944\n",
      "Epoch 21/35\n",
      "2s - loss: 1.0203 - acc: 0.5998 - val_loss: 1.0298 - val_acc: 0.6001\n",
      "Epoch 22/35\n",
      "2s - loss: 1.0180 - acc: 0.6012 - val_loss: 1.0269 - val_acc: 0.6006\n",
      "Epoch 23/35\n",
      "2s - loss: 1.0156 - acc: 0.6024 - val_loss: 1.0251 - val_acc: 0.6022\n",
      "Epoch 24/35\n",
      "2s - loss: 1.0132 - acc: 0.6042 - val_loss: 1.0340 - val_acc: 0.5954\n",
      "Epoch 25/35\n",
      "2s - loss: 1.0115 - acc: 0.6042 - val_loss: 1.0241 - val_acc: 0.6021\n",
      "Epoch 26/35\n",
      "2s - loss: 1.0095 - acc: 0.6052 - val_loss: 1.0229 - val_acc: 0.6047\n",
      "Epoch 27/35\n",
      "2s - loss: 1.0088 - acc: 0.6042 - val_loss: 1.0250 - val_acc: 0.6031\n",
      "Epoch 28/35\n",
      "2s - loss: 1.0060 - acc: 0.6060 - val_loss: 1.0249 - val_acc: 0.6027\n",
      "Epoch 29/35\n",
      "2s - loss: 1.0044 - acc: 0.6072 - val_loss: 1.0217 - val_acc: 0.6056\n",
      "Epoch 30/35\n",
      "2s - loss: 1.0030 - acc: 0.6070 - val_loss: 1.0216 - val_acc: 0.6049\n",
      "Epoch 31/35\n",
      "2s - loss: 1.0005 - acc: 0.6074 - val_loss: 1.0197 - val_acc: 0.6054\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9999 - acc: 0.6103 - val_loss: 1.0209 - val_acc: 0.6049\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9974 - acc: 0.6113 - val_loss: 1.0184 - val_acc: 0.6051\n",
      "Epoch 34/35\n",
      "2s - loss: 0.9947 - acc: 0.6128 - val_loss: 1.0225 - val_acc: 0.6034\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9935 - acc: 0.6114 - val_loss: 1.0229 - val_acc: 0.6054\n",
      "19936/20000 [============================>.] - ETA: 0sva acc: 0.60225\n",
      "te acc: 0.60535\n",
      "2017-01-12 21:06:00.172173 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 1.1219 - acc: 0.5589 - val_loss: 1.0491 - val_acc: 0.5864\n",
      "Epoch 2/35\n",
      "2s - loss: 1.0564 - acc: 0.5876 - val_loss: 1.0435 - val_acc: 0.5977\n",
      "Epoch 3/35\n",
      "2s - loss: 1.0522 - acc: 0.5897 - val_loss: 1.0383 - val_acc: 0.5965\n",
      "Epoch 4/35\n",
      "2s - loss: 1.0498 - acc: 0.5903 - val_loss: 1.0382 - val_acc: 0.5942\n",
      "Epoch 5/35\n",
      "2s - loss: 1.0479 - acc: 0.5908 - val_loss: 1.0402 - val_acc: 0.5946\n",
      "Epoch 6/35\n",
      "2s - loss: 1.0457 - acc: 0.5919 - val_loss: 1.0449 - val_acc: 0.5865\n",
      "Epoch 7/35\n",
      "3s - loss: 1.0443 - acc: 0.5920 - val_loss: 1.0387 - val_acc: 0.5979\n",
      "Epoch 8/35\n",
      "3s - loss: 1.0430 - acc: 0.5927 - val_loss: 1.0396 - val_acc: 0.5967\n",
      "Epoch 9/35\n",
      "2s - loss: 1.0415 - acc: 0.5927 - val_loss: 1.0392 - val_acc: 0.5964\n",
      "Epoch 10/35\n",
      "2s - loss: 1.0402 - acc: 0.5949 - val_loss: 1.0386 - val_acc: 0.5935\n",
      "Epoch 11/35\n",
      "2s - loss: 1.0387 - acc: 0.5943 - val_loss: 1.0350 - val_acc: 0.5949\n",
      "Epoch 12/35\n",
      "2s - loss: 1.0375 - acc: 0.5948 - val_loss: 1.0435 - val_acc: 0.5914\n",
      "Epoch 13/35\n",
      "2s - loss: 1.0355 - acc: 0.5961 - val_loss: 1.0324 - val_acc: 0.5976\n",
      "Epoch 14/35\n",
      "2s - loss: 1.0328 - acc: 0.5964 - val_loss: 1.0319 - val_acc: 0.5994\n",
      "Epoch 15/35\n",
      "2s - loss: 1.0314 - acc: 0.5975 - val_loss: 1.0340 - val_acc: 0.5978\n",
      "Epoch 16/35\n",
      "2s - loss: 1.0294 - acc: 0.5985 - val_loss: 1.0313 - val_acc: 0.5940\n",
      "Epoch 17/35\n",
      "2s - loss: 1.0281 - acc: 0.5992 - val_loss: 1.0296 - val_acc: 0.5995\n",
      "Epoch 18/35\n",
      "2s - loss: 1.0252 - acc: 0.5991 - val_loss: 1.0276 - val_acc: 0.6000\n",
      "Epoch 19/35\n",
      "2s - loss: 1.0238 - acc: 0.6006 - val_loss: 1.0283 - val_acc: 0.6030\n",
      "Epoch 20/35\n",
      "2s - loss: 1.0216 - acc: 0.6014 - val_loss: 1.0287 - val_acc: 0.5968\n",
      "Epoch 21/35\n",
      "2s - loss: 1.0192 - acc: 0.6015 - val_loss: 1.0247 - val_acc: 0.6007\n",
      "Epoch 22/35\n",
      "3s - loss: 1.0173 - acc: 0.6020 - val_loss: 1.0221 - val_acc: 0.6006\n",
      "Epoch 23/35\n",
      "3s - loss: 1.0157 - acc: 0.6039 - val_loss: 1.0236 - val_acc: 0.6014\n",
      "Epoch 24/35\n",
      "2s - loss: 1.0134 - acc: 0.6035 - val_loss: 1.0280 - val_acc: 0.6035\n",
      "Epoch 25/35\n",
      "2s - loss: 1.0114 - acc: 0.6054 - val_loss: 1.0271 - val_acc: 0.6012\n",
      "Epoch 26/35\n",
      "2s - loss: 1.0094 - acc: 0.6055 - val_loss: 1.0203 - val_acc: 0.6036\n",
      "Epoch 27/35\n",
      "2s - loss: 1.0076 - acc: 0.6071 - val_loss: 1.0190 - val_acc: 0.6033\n",
      "Epoch 28/35\n",
      "2s - loss: 1.0068 - acc: 0.6067 - val_loss: 1.0195 - val_acc: 0.6042\n",
      "Epoch 29/35\n",
      "2s - loss: 1.0052 - acc: 0.6074 - val_loss: 1.0198 - val_acc: 0.6038\n",
      "Epoch 30/35\n",
      "2s - loss: 1.0024 - acc: 0.6088 - val_loss: 1.0174 - val_acc: 0.6045\n",
      "Epoch 31/35\n",
      "2s - loss: 1.0009 - acc: 0.6088 - val_loss: 1.0208 - val_acc: 0.6027\n",
      "Epoch 32/35\n",
      "2s - loss: 0.9981 - acc: 0.6103 - val_loss: 1.0180 - val_acc: 0.6028\n",
      "Epoch 33/35\n",
      "2s - loss: 0.9977 - acc: 0.6094 - val_loss: 1.0138 - val_acc: 0.6073\n",
      "Epoch 34/35\n",
      "2s - loss: 0.9957 - acc: 0.6123 - val_loss: 1.0141 - val_acc: 0.6068\n",
      "Epoch 35/35\n",
      "2s - loss: 0.9947 - acc: 0.6110 - val_loss: 1.0152 - val_acc: 0.6069\n",
      "19936/20000 [============================>.] - ETA: 0sva acc: 0.59975\n",
      "te acc: 0.60685\n",
      "2017-01-12 21:07:39.385249 stack:1/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4478 - acc: 0.8167 - val_loss: 0.4342 - val_acc: 0.8283\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4319 - acc: 0.8296 - val_loss: 0.4305 - val_acc: 0.8323\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4310 - acc: 0.8305 - val_loss: 0.4311 - val_acc: 0.8323\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4294 - acc: 0.8314 - val_loss: 0.4293 - val_acc: 0.8327\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4287 - acc: 0.8312 - val_loss: 0.4329 - val_acc: 0.8336\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4279 - acc: 0.8310 - val_loss: 0.4309 - val_acc: 0.8327\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4271 - acc: 0.8328 - val_loss: 0.4296 - val_acc: 0.8316\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4264 - acc: 0.8323 - val_loss: 0.4288 - val_acc: 0.8346\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4257 - acc: 0.8332 - val_loss: 0.4295 - val_acc: 0.8347\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4252 - acc: 0.8332 - val_loss: 0.4283 - val_acc: 0.8338\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4248 - acc: 0.8329 - val_loss: 0.4280 - val_acc: 0.8339\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4241 - acc: 0.8336 - val_loss: 0.4292 - val_acc: 0.8347\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4237 - acc: 0.8334 - val_loss: 0.4281 - val_acc: 0.8327\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4231 - acc: 0.8333 - val_loss: 0.4293 - val_acc: 0.8347\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4217 - acc: 0.8337 - val_loss: 0.4278 - val_acc: 0.8327\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4219 - acc: 0.8343 - val_loss: 0.4302 - val_acc: 0.8354\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4211 - acc: 0.8336 - val_loss: 0.4274 - val_acc: 0.8350\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4202 - acc: 0.8342 - val_loss: 0.4258 - val_acc: 0.8344\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4193 - acc: 0.8349 - val_loss: 0.4269 - val_acc: 0.8359\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4186 - acc: 0.8350 - val_loss: 0.4250 - val_acc: 0.8358\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4185 - acc: 0.8350 - val_loss: 0.4261 - val_acc: 0.8366\n",
      "Epoch 22/35\n",
      "2s - loss: 0.4179 - acc: 0.8349 - val_loss: 0.4274 - val_acc: 0.8370\n",
      "Epoch 23/35\n",
      "2s - loss: 0.4173 - acc: 0.8350 - val_loss: 0.4251 - val_acc: 0.8358\n",
      "Epoch 24/35\n",
      "2s - loss: 0.4166 - acc: 0.8360 - val_loss: 0.4245 - val_acc: 0.8358\n",
      "Epoch 25/35\n",
      "2s - loss: 0.4161 - acc: 0.8352 - val_loss: 0.4232 - val_acc: 0.8348\n",
      "Epoch 26/35\n",
      "2s - loss: 0.4149 - acc: 0.8362 - val_loss: 0.4243 - val_acc: 0.8351\n",
      "Epoch 27/35\n",
      "2s - loss: 0.4149 - acc: 0.8356 - val_loss: 0.4238 - val_acc: 0.8346\n",
      "Epoch 28/35\n",
      "2s - loss: 0.4136 - acc: 0.8372 - val_loss: 0.4235 - val_acc: 0.8359\n",
      "Epoch 29/35\n",
      "2s - loss: 0.4130 - acc: 0.8367 - val_loss: 0.4228 - val_acc: 0.8364\n",
      "Epoch 30/35\n",
      "2s - loss: 0.4130 - acc: 0.8366 - val_loss: 0.4247 - val_acc: 0.8367\n",
      "Epoch 31/35\n",
      "2s - loss: 0.4115 - acc: 0.8378 - val_loss: 0.4237 - val_acc: 0.8359\n",
      "Epoch 32/35\n",
      "2s - loss: 0.4115 - acc: 0.8374 - val_loss: 0.4221 - val_acc: 0.8367\n",
      "Epoch 33/35\n",
      "2s - loss: 0.4106 - acc: 0.8375 - val_loss: 0.4215 - val_acc: 0.8361\n",
      "Epoch 34/35\n",
      "2s - loss: 0.4096 - acc: 0.8378 - val_loss: 0.4225 - val_acc: 0.8350\n",
      "Epoch 35/35\n",
      "2s - loss: 0.4087 - acc: 0.8380 - val_loss: 0.4226 - val_acc: 0.8371\n",
      "19840/20000 [============================>.] - ETA: 0sva acc: 0.834375\n",
      "te acc: 0.8371\n",
      "2017-01-12 21:09:12.722037 stack:2/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4501 - acc: 0.8171 - val_loss: 0.4342 - val_acc: 0.8337\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4313 - acc: 0.8306 - val_loss: 0.4321 - val_acc: 0.8306\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4305 - acc: 0.8315 - val_loss: 0.4306 - val_acc: 0.8333\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4291 - acc: 0.8326 - val_loss: 0.4301 - val_acc: 0.8342\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4283 - acc: 0.8322 - val_loss: 0.4303 - val_acc: 0.8337\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4272 - acc: 0.8320 - val_loss: 0.4291 - val_acc: 0.8321\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4268 - acc: 0.8336 - val_loss: 0.4287 - val_acc: 0.8337\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4260 - acc: 0.8338 - val_loss: 0.4294 - val_acc: 0.8345\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4255 - acc: 0.8333 - val_loss: 0.4293 - val_acc: 0.8312\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4251 - acc: 0.8338 - val_loss: 0.4303 - val_acc: 0.8338\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4243 - acc: 0.8330 - val_loss: 0.4285 - val_acc: 0.8347\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4239 - acc: 0.8342 - val_loss: 0.4278 - val_acc: 0.8330\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4234 - acc: 0.8340 - val_loss: 0.4298 - val_acc: 0.8347\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4229 - acc: 0.8340 - val_loss: 0.4277 - val_acc: 0.8333\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4219 - acc: 0.8348 - val_loss: 0.4270 - val_acc: 0.8339\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4218 - acc: 0.8353 - val_loss: 0.4277 - val_acc: 0.8350\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4204 - acc: 0.8352 - val_loss: 0.4259 - val_acc: 0.8348\n",
      "Epoch 18/35\n",
      "3s - loss: 0.4201 - acc: 0.8361 - val_loss: 0.4261 - val_acc: 0.8358\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4193 - acc: 0.8361 - val_loss: 0.4283 - val_acc: 0.8310\n",
      "Epoch 20/35\n",
      "3s - loss: 0.4186 - acc: 0.8353 - val_loss: 0.4270 - val_acc: 0.8335\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4183 - acc: 0.8358 - val_loss: 0.4245 - val_acc: 0.8341\n",
      "Epoch 22/35\n",
      "2s - loss: 0.4174 - acc: 0.8354 - val_loss: 0.4244 - val_acc: 0.8345\n",
      "Epoch 23/35\n",
      "2s - loss: 0.4170 - acc: 0.8353 - val_loss: 0.4250 - val_acc: 0.8357\n",
      "Epoch 24/35\n",
      "2s - loss: 0.4164 - acc: 0.8360 - val_loss: 0.4258 - val_acc: 0.8326\n",
      "Epoch 25/35\n",
      "2s - loss: 0.4158 - acc: 0.8366 - val_loss: 0.4244 - val_acc: 0.8357\n",
      "Epoch 26/35\n",
      "3s - loss: 0.4148 - acc: 0.8369 - val_loss: 0.4232 - val_acc: 0.8353\n",
      "Epoch 27/35\n",
      "3s - loss: 0.4141 - acc: 0.8368 - val_loss: 0.4241 - val_acc: 0.8360\n",
      "Epoch 28/35\n",
      "2s - loss: 0.4137 - acc: 0.8371 - val_loss: 0.4230 - val_acc: 0.8355\n",
      "Epoch 29/35\n",
      "2s - loss: 0.4132 - acc: 0.8367 - val_loss: 0.4228 - val_acc: 0.8347\n",
      "Epoch 30/35\n",
      "2s - loss: 0.4124 - acc: 0.8379 - val_loss: 0.4225 - val_acc: 0.8360\n",
      "Epoch 31/35\n",
      "2s - loss: 0.4124 - acc: 0.8379 - val_loss: 0.4232 - val_acc: 0.8365\n",
      "Epoch 32/35\n",
      "2s - loss: 0.4113 - acc: 0.8381 - val_loss: 0.4232 - val_acc: 0.8350\n",
      "Epoch 33/35\n",
      "2s - loss: 0.4100 - acc: 0.8393 - val_loss: 0.4222 - val_acc: 0.8361\n",
      "Epoch 34/35\n",
      "2s - loss: 0.4096 - acc: 0.8385 - val_loss: 0.4223 - val_acc: 0.8376\n",
      "Epoch 35/35\n",
      "2s - loss: 0.4093 - acc: 0.8385 - val_loss: 0.4215 - val_acc: 0.8366\n",
      "19872/20000 [============================>.] - ETA: 0sva acc: 0.8336875\n",
      "te acc: 0.83655\n",
      "2017-01-12 21:10:52.410318 stack:3/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4506 - acc: 0.8152 - val_loss: 0.4303 - val_acc: 0.8335\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4320 - acc: 0.8296 - val_loss: 0.4310 - val_acc: 0.8326\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4307 - acc: 0.8298 - val_loss: 0.4302 - val_acc: 0.8334\n",
      "Epoch 4/35\n",
      "3s - loss: 0.4294 - acc: 0.8309 - val_loss: 0.4304 - val_acc: 0.8327\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4287 - acc: 0.8322 - val_loss: 0.4293 - val_acc: 0.8345\n",
      "Epoch 6/35\n",
      "3s - loss: 0.4276 - acc: 0.8321 - val_loss: 0.4296 - val_acc: 0.8342\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4273 - acc: 0.8320 - val_loss: 0.4285 - val_acc: 0.8338\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4264 - acc: 0.8343 - val_loss: 0.4288 - val_acc: 0.8336\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4258 - acc: 0.8326 - val_loss: 0.4281 - val_acc: 0.8338\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4251 - acc: 0.8328 - val_loss: 0.4281 - val_acc: 0.8350\n",
      "Epoch 11/35\n",
      "3s - loss: 0.4244 - acc: 0.8337 - val_loss: 0.4273 - val_acc: 0.8343\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4240 - acc: 0.8329 - val_loss: 0.4278 - val_acc: 0.8348\n",
      "Epoch 13/35\n",
      "3s - loss: 0.4231 - acc: 0.8343 - val_loss: 0.4274 - val_acc: 0.8356\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4224 - acc: 0.8344 - val_loss: 0.4290 - val_acc: 0.8309\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4218 - acc: 0.8331 - val_loss: 0.4267 - val_acc: 0.8336\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4213 - acc: 0.8345 - val_loss: 0.4279 - val_acc: 0.8314\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4202 - acc: 0.8343 - val_loss: 0.4255 - val_acc: 0.8338\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4199 - acc: 0.8343 - val_loss: 0.4256 - val_acc: 0.8357\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4187 - acc: 0.8357 - val_loss: 0.4265 - val_acc: 0.8334\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4188 - acc: 0.8351 - val_loss: 0.4254 - val_acc: 0.8335\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4172 - acc: 0.8356 - val_loss: 0.4254 - val_acc: 0.8355\n",
      "Epoch 22/35\n",
      "3s - loss: 0.4166 - acc: 0.8358 - val_loss: 0.4247 - val_acc: 0.8360\n",
      "Epoch 23/35\n",
      "2s - loss: 0.4158 - acc: 0.8368 - val_loss: 0.4245 - val_acc: 0.8352\n",
      "Epoch 24/35\n",
      "2s - loss: 0.4153 - acc: 0.8356 - val_loss: 0.4251 - val_acc: 0.8361\n",
      "Epoch 25/35\n",
      "2s - loss: 0.4148 - acc: 0.8367 - val_loss: 0.4240 - val_acc: 0.8365\n",
      "Epoch 26/35\n",
      "2s - loss: 0.4138 - acc: 0.8372 - val_loss: 0.4247 - val_acc: 0.8367\n",
      "Epoch 27/35\n",
      "2s - loss: 0.4132 - acc: 0.8368 - val_loss: 0.4226 - val_acc: 0.8365\n",
      "Epoch 28/35\n",
      "2s - loss: 0.4131 - acc: 0.8370 - val_loss: 0.4219 - val_acc: 0.8347\n",
      "Epoch 29/35\n",
      "2s - loss: 0.4113 - acc: 0.8375 - val_loss: 0.4218 - val_acc: 0.8355\n",
      "Epoch 30/35\n",
      "2s - loss: 0.4113 - acc: 0.8379 - val_loss: 0.4220 - val_acc: 0.8348\n",
      "Epoch 31/35\n",
      "2s - loss: 0.4105 - acc: 0.8381 - val_loss: 0.4228 - val_acc: 0.8358\n",
      "Epoch 32/35\n",
      "2s - loss: 0.4096 - acc: 0.8381 - val_loss: 0.4215 - val_acc: 0.8357\n",
      "Epoch 33/35\n",
      "2s - loss: 0.4083 - acc: 0.8392 - val_loss: 0.4200 - val_acc: 0.8364\n",
      "Epoch 34/35\n",
      "2s - loss: 0.4079 - acc: 0.8387 - val_loss: 0.4205 - val_acc: 0.8372\n",
      "Epoch 35/35\n",
      "2s - loss: 0.4067 - acc: 0.8397 - val_loss: 0.4216 - val_acc: 0.8360\n",
      "19744/20000 [============================>.] - ETA: 0sva acc: 0.832375\n",
      "te acc: 0.836\n",
      "2017-01-12 21:12:33.028205 stack:4/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4499 - acc: 0.8156 - val_loss: 0.4350 - val_acc: 0.8333\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4327 - acc: 0.8288 - val_loss: 0.4296 - val_acc: 0.8339\n",
      "Epoch 3/35\n",
      "2s - loss: 0.4314 - acc: 0.8299 - val_loss: 0.4329 - val_acc: 0.8337\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4302 - acc: 0.8301 - val_loss: 0.4346 - val_acc: 0.8337\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4294 - acc: 0.8317 - val_loss: 0.4297 - val_acc: 0.8340\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4286 - acc: 0.8311 - val_loss: 0.4295 - val_acc: 0.8328\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4277 - acc: 0.8322 - val_loss: 0.4315 - val_acc: 0.8292\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4267 - acc: 0.8325 - val_loss: 0.4292 - val_acc: 0.8326\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4261 - acc: 0.8320 - val_loss: 0.4289 - val_acc: 0.8325\n",
      "Epoch 10/35\n",
      "3s - loss: 0.4253 - acc: 0.8329 - val_loss: 0.4300 - val_acc: 0.8356\n",
      "Epoch 11/35\n",
      "3s - loss: 0.4253 - acc: 0.8336 - val_loss: 0.4296 - val_acc: 0.8326\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4242 - acc: 0.8335 - val_loss: 0.4282 - val_acc: 0.8340\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4238 - acc: 0.8335 - val_loss: 0.4274 - val_acc: 0.8344\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4232 - acc: 0.8334 - val_loss: 0.4264 - val_acc: 0.8351\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4224 - acc: 0.8335 - val_loss: 0.4266 - val_acc: 0.8350\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4218 - acc: 0.8348 - val_loss: 0.4255 - val_acc: 0.8346\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4206 - acc: 0.8348 - val_loss: 0.4258 - val_acc: 0.8346\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4201 - acc: 0.8337 - val_loss: 0.4261 - val_acc: 0.8336\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4195 - acc: 0.8344 - val_loss: 0.4272 - val_acc: 0.8360\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4192 - acc: 0.8354 - val_loss: 0.4255 - val_acc: 0.8342\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4185 - acc: 0.8353 - val_loss: 0.4245 - val_acc: 0.8343\n",
      "Epoch 22/35\n",
      "2s - loss: 0.4180 - acc: 0.8351 - val_loss: 0.4257 - val_acc: 0.8339\n",
      "Epoch 23/35\n",
      "2s - loss: 0.4175 - acc: 0.8350 - val_loss: 0.4245 - val_acc: 0.8348\n",
      "Epoch 24/35\n",
      "2s - loss: 0.4160 - acc: 0.8359 - val_loss: 0.4240 - val_acc: 0.8341\n",
      "Epoch 25/35\n",
      "2s - loss: 0.4152 - acc: 0.8358 - val_loss: 0.4276 - val_acc: 0.8368\n",
      "Epoch 26/35\n",
      "2s - loss: 0.4153 - acc: 0.8362 - val_loss: 0.4232 - val_acc: 0.8340\n",
      "Epoch 27/35\n",
      "2s - loss: 0.4140 - acc: 0.8370 - val_loss: 0.4233 - val_acc: 0.8358\n",
      "Epoch 28/35\n",
      "2s - loss: 0.4138 - acc: 0.8367 - val_loss: 0.4230 - val_acc: 0.8353\n",
      "Epoch 29/35\n",
      "2s - loss: 0.4132 - acc: 0.8356 - val_loss: 0.4242 - val_acc: 0.8335\n",
      "Epoch 30/35\n",
      "3s - loss: 0.4126 - acc: 0.8378 - val_loss: 0.4228 - val_acc: 0.8342\n",
      "Epoch 31/35\n",
      "3s - loss: 0.4116 - acc: 0.8378 - val_loss: 0.4231 - val_acc: 0.8347\n",
      "Epoch 32/35\n",
      "3s - loss: 0.4112 - acc: 0.8381 - val_loss: 0.4231 - val_acc: 0.8373\n",
      "Epoch 33/35\n",
      "2s - loss: 0.4104 - acc: 0.8377 - val_loss: 0.4225 - val_acc: 0.8349\n",
      "Epoch 34/35\n",
      "2s - loss: 0.4101 - acc: 0.8374 - val_loss: 0.4240 - val_acc: 0.8346\n",
      "Epoch 35/35\n",
      "2s - loss: 0.4095 - acc: 0.8392 - val_loss: 0.4212 - val_acc: 0.8360\n",
      "19872/20000 [============================>.] - ETA: 0sva acc: 0.8364375\n",
      "te acc: 0.836\n",
      "2017-01-12 21:14:10.372062 stack:5/5\n",
      "Train on 64000 samples, validate on 20000 samples\n",
      "Epoch 1/35\n",
      "2s - loss: 0.4517 - acc: 0.8160 - val_loss: 0.4330 - val_acc: 0.8273\n",
      "Epoch 2/35\n",
      "2s - loss: 0.4333 - acc: 0.8288 - val_loss: 0.4310 - val_acc: 0.8319\n",
      "Epoch 3/35\n",
      "3s - loss: 0.4322 - acc: 0.8295 - val_loss: 0.4325 - val_acc: 0.8333\n",
      "Epoch 4/35\n",
      "2s - loss: 0.4316 - acc: 0.8305 - val_loss: 0.4311 - val_acc: 0.8309\n",
      "Epoch 5/35\n",
      "2s - loss: 0.4303 - acc: 0.8311 - val_loss: 0.4313 - val_acc: 0.8336\n",
      "Epoch 6/35\n",
      "2s - loss: 0.4296 - acc: 0.8301 - val_loss: 0.4327 - val_acc: 0.8341\n",
      "Epoch 7/35\n",
      "2s - loss: 0.4293 - acc: 0.8308 - val_loss: 0.4306 - val_acc: 0.8345\n",
      "Epoch 8/35\n",
      "2s - loss: 0.4286 - acc: 0.8319 - val_loss: 0.4317 - val_acc: 0.8343\n",
      "Epoch 9/35\n",
      "2s - loss: 0.4279 - acc: 0.8320 - val_loss: 0.4292 - val_acc: 0.8348\n",
      "Epoch 10/35\n",
      "2s - loss: 0.4275 - acc: 0.8315 - val_loss: 0.4292 - val_acc: 0.8313\n",
      "Epoch 11/35\n",
      "2s - loss: 0.4265 - acc: 0.8317 - val_loss: 0.4282 - val_acc: 0.8347\n",
      "Epoch 12/35\n",
      "2s - loss: 0.4261 - acc: 0.8321 - val_loss: 0.4280 - val_acc: 0.8341\n",
      "Epoch 13/35\n",
      "2s - loss: 0.4259 - acc: 0.8319 - val_loss: 0.4271 - val_acc: 0.8350\n",
      "Epoch 14/35\n",
      "2s - loss: 0.4250 - acc: 0.8319 - val_loss: 0.4279 - val_acc: 0.8350\n",
      "Epoch 15/35\n",
      "2s - loss: 0.4246 - acc: 0.8322 - val_loss: 0.4279 - val_acc: 0.8347\n",
      "Epoch 16/35\n",
      "2s - loss: 0.4234 - acc: 0.8331 - val_loss: 0.4263 - val_acc: 0.8349\n",
      "Epoch 17/35\n",
      "2s - loss: 0.4235 - acc: 0.8332 - val_loss: 0.4260 - val_acc: 0.8345\n",
      "Epoch 18/35\n",
      "2s - loss: 0.4227 - acc: 0.8329 - val_loss: 0.4254 - val_acc: 0.8345\n",
      "Epoch 19/35\n",
      "2s - loss: 0.4221 - acc: 0.8327 - val_loss: 0.4253 - val_acc: 0.8341\n",
      "Epoch 20/35\n",
      "2s - loss: 0.4211 - acc: 0.8341 - val_loss: 0.4299 - val_acc: 0.8288\n",
      "Epoch 21/35\n",
      "2s - loss: 0.4207 - acc: 0.8337 - val_loss: 0.4251 - val_acc: 0.8352\n",
      "Epoch 22/35\n",
      "2s - loss: 0.4200 - acc: 0.8341 - val_loss: 0.4242 - val_acc: 0.8347\n",
      "Epoch 23/35\n",
      "2s - loss: 0.4195 - acc: 0.8341 - val_loss: 0.4245 - val_acc: 0.8346\n",
      "Epoch 24/35\n",
      "2s - loss: 0.4187 - acc: 0.8337 - val_loss: 0.4273 - val_acc: 0.8314\n",
      "Epoch 25/35\n",
      "2s - loss: 0.4176 - acc: 0.8346 - val_loss: 0.4239 - val_acc: 0.8353\n",
      "Epoch 26/35\n",
      "2s - loss: 0.4179 - acc: 0.8345 - val_loss: 0.4230 - val_acc: 0.8361\n",
      "Epoch 27/35\n",
      "2s - loss: 0.4166 - acc: 0.8355 - val_loss: 0.4236 - val_acc: 0.8341\n",
      "Epoch 28/35\n",
      "2s - loss: 0.4163 - acc: 0.8353 - val_loss: 0.4230 - val_acc: 0.8360\n",
      "Epoch 29/35\n",
      "2s - loss: 0.4154 - acc: 0.8358 - val_loss: 0.4227 - val_acc: 0.8360\n",
      "Epoch 30/35\n",
      "2s - loss: 0.4148 - acc: 0.8357 - val_loss: 0.4219 - val_acc: 0.8371\n",
      "Epoch 31/35\n",
      "2s - loss: 0.4142 - acc: 0.8364 - val_loss: 0.4218 - val_acc: 0.8359\n",
      "Epoch 32/35\n",
      "2s - loss: 0.4136 - acc: 0.8369 - val_loss: 0.4224 - val_acc: 0.8358\n",
      "Epoch 33/35\n",
      "2s - loss: 0.4124 - acc: 0.8369 - val_loss: 0.4218 - val_acc: 0.8371\n",
      "Epoch 34/35\n",
      "2s - loss: 0.4117 - acc: 0.8380 - val_loss: 0.4223 - val_acc: 0.8364\n",
      "Epoch 35/35\n",
      "2s - loss: 0.4113 - acc: 0.8376 - val_loss: 0.4208 - val_acc: 0.8365\n",
      "19968/20000 [============================>.] - ETA: 0sva acc: 0.8396875\n",
      "te acc: 0.8365\n",
      "2017-01-12 21:15:47.652330 save dmd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "'''dm-nn stack for education/age/gender'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "import cfg\n",
    "\n",
    "#-----------------------myfunc-----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "#-----------------------load dataset----------------------\n",
    "df_all = pd.read_csv(cfg.data_path + 'all.csv',encoding='utf8',usecols=['Id','Education','age','gender'],nrows=100000)\n",
    "ys = {}\n",
    "for label in ['Education','age','gender']:\n",
    "    ys[label] = np.array(df_all[label])\n",
    "    \n",
    "model = Doc2Vec.load(cfg.data_path + 'dm_d2v.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(100000)])\n",
    "\n",
    "#----------------------dmd2v stack for Education/age/gender---------------------------\n",
    "\n",
    "df_stack = pd.DataFrame(index=range(len(df_all)))\n",
    "TR = 80000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "\n",
    "feat = 'dmd2v'\n",
    "for i,lb in enumerate(['Education','age','gender']):\n",
    "    num_class = len(pd.value_counts(ys[lb]))\n",
    "    y = ys[lb][:TR]\n",
    "    y_te = ys[lb][TR:]\n",
    "    \n",
    "    stack = np.zeros((X.shape[0],num_class))\n",
    "    stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "    \n",
    "    for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "        print('{} stack:{}/{}'.format(datetime.now(),k+1,n))\n",
    "        nb_classes = num_class\n",
    "        X_train = X[tr]\n",
    "        y_train = y[tr]\n",
    "        X_test = X_te\n",
    "        y_test = y_te\n",
    "\n",
    "        X_train = X_train.astype('float32')\n",
    "        X_test = X_test.astype('float32')\n",
    "        Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "        Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "\n",
    "        model = Sequential()\n",
    "        model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "        model.add(Dropout(0.1))\n",
    "        model.add(Activation('tanh'))\n",
    "        model.add(Dense(nb_classes))\n",
    "        model.add(Activation('softmax'))\n",
    "\n",
    "        model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "        history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=35,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "        y_pred_va = model.predict_proba(X[va])\n",
    "        y_pred_te = model.predict_proba(X_te)\n",
    "        print('va acc:',myAcc(y[va],y_pred_va))\n",
    "        print('te acc:',myAcc(y_te,y_pred_te))\n",
    "        stack[va] += y_pred_va\n",
    "        stack_te += y_pred_te\n",
    "    stack_te /= n\n",
    "    stack_all = np.vstack([stack,stack_te])\n",
    "    for l in range(stack_all.shape[1]):\n",
    "        df_stack['{}_{}_{}'.format(feat,lb,l)] = stack_all[:,l]\n",
    "df_stack.to_csv(cfg.data_path + 'dmd2v_stack_10W.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dmd2v stack done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Education\n",
      "Index(['tfidf_Education_0', 'tfidf_Education_1', 'tfidf_Education_2',\n",
      "       'tfidf_Education_3', 'tfidf_Education_4', 'tfidf_Education_5',\n",
      "       'tfidf_age_0', 'tfidf_age_1', 'tfidf_age_2', 'tfidf_age_3',\n",
      "       'tfidf_age_4', 'tfidf_age_5', 'tfidf_gender_0', 'tfidf_gender_1',\n",
      "       'dbowd2v_Education_0', 'dbowd2v_Education_1', 'dbowd2v_Education_2',\n",
      "       'dbowd2v_Education_3', 'dbowd2v_Education_4', 'dbowd2v_Education_5',\n",
      "       'dbowd2v_age_0', 'dbowd2v_age_1', 'dbowd2v_age_2', 'dbowd2v_age_3',\n",
      "       'dbowd2v_age_4', 'dbowd2v_age_5', 'dbowd2v_gender_0',\n",
      "       'dbowd2v_gender_1', 'dmd2v_Education_0', 'dmd2v_Education_1',\n",
      "       'dmd2v_Education_2', 'dmd2v_Education_3', 'dmd2v_Education_4',\n",
      "       'dmd2v_Education_5', 'dmd2v_age_0', 'dmd2v_age_1', 'dmd2v_age_2',\n",
      "       'dmd2v_age_3', 'dmd2v_age_4', 'dmd2v_age_5', 'dmd2v_gender_0',\n",
      "       'dmd2v_gender_1'],\n",
      "      dtype='object')\n",
      "[0]\ttrain-merror:0.315875\teval-merror:0.34185\ttrain-acc:0.684125\teval-acc:0.65815\n",
      "Multiple eval metrics have been passed: 'eval-acc' will be used for early stopping.\n",
      "\n",
      "Will train until eval-acc hasn't improved in 25 rounds.\n",
      "[1]\ttrain-merror:0.312562\teval-merror:0.3326\ttrain-acc:0.687438\teval-acc:0.6674\n",
      "[2]\ttrain-merror:0.309487\teval-merror:0.33145\ttrain-acc:0.690512\teval-acc:0.66855\n",
      "[3]\ttrain-merror:0.3089\teval-merror:0.3319\ttrain-acc:0.6911\teval-acc:0.6681\n",
      "[4]\ttrain-merror:0.308037\teval-merror:0.331\ttrain-acc:0.691963\teval-acc:0.669\n",
      "[5]\ttrain-merror:0.30785\teval-merror:0.3307\ttrain-acc:0.69215\teval-acc:0.6693\n",
      "[6]\ttrain-merror:0.306888\teval-merror:0.3308\ttrain-acc:0.693113\teval-acc:0.6692\n",
      "[7]\ttrain-merror:0.306863\teval-merror:0.33095\ttrain-acc:0.693137\teval-acc:0.66905\n",
      "[8]\ttrain-merror:0.306675\teval-merror:0.33015\ttrain-acc:0.693325\teval-acc:0.66985\n",
      "[9]\ttrain-merror:0.306713\teval-merror:0.3314\ttrain-acc:0.693287\teval-acc:0.6686\n",
      "[10]\ttrain-merror:0.306475\teval-merror:0.33095\ttrain-acc:0.693525\teval-acc:0.66905\n",
      "[11]\ttrain-merror:0.306313\teval-merror:0.33155\ttrain-acc:0.693688\teval-acc:0.66845\n",
      "[12]\ttrain-merror:0.306125\teval-merror:0.3315\ttrain-acc:0.693875\teval-acc:0.6685\n",
      "[13]\ttrain-merror:0.305712\teval-merror:0.3308\ttrain-acc:0.694287\teval-acc:0.6692\n",
      "[14]\ttrain-merror:0.30575\teval-merror:0.3303\ttrain-acc:0.69425\teval-acc:0.6697\n",
      "[15]\ttrain-merror:0.305788\teval-merror:0.33035\ttrain-acc:0.694213\teval-acc:0.66965\n",
      "[16]\ttrain-merror:0.305675\teval-merror:0.33015\ttrain-acc:0.694325\teval-acc:0.66985\n",
      "[17]\ttrain-merror:0.305175\teval-merror:0.3301\ttrain-acc:0.694825\teval-acc:0.6699\n",
      "[18]\ttrain-merror:0.304975\teval-merror:0.33005\ttrain-acc:0.695025\teval-acc:0.66995\n",
      "[19]\ttrain-merror:0.304875\teval-merror:0.32965\ttrain-acc:0.695125\teval-acc:0.67035\n",
      "[20]\ttrain-merror:0.304787\teval-merror:0.32965\ttrain-acc:0.695213\teval-acc:0.67035\n",
      "[21]\ttrain-merror:0.304725\teval-merror:0.3297\ttrain-acc:0.695275\teval-acc:0.6703\n",
      "[22]\ttrain-merror:0.304763\teval-merror:0.3295\ttrain-acc:0.695237\teval-acc:0.6705\n",
      "[23]\ttrain-merror:0.304875\teval-merror:0.3298\ttrain-acc:0.695125\teval-acc:0.6702\n",
      "[24]\ttrain-merror:0.304738\teval-merror:0.3296\ttrain-acc:0.695263\teval-acc:0.6704\n",
      "[25]\ttrain-merror:0.30465\teval-merror:0.3299\ttrain-acc:0.69535\teval-acc:0.6701\n",
      "[26]\ttrain-merror:0.304662\teval-merror:0.3296\ttrain-acc:0.695338\teval-acc:0.6704\n",
      "[27]\ttrain-merror:0.30445\teval-merror:0.32975\ttrain-acc:0.69555\teval-acc:0.67025\n",
      "[28]\ttrain-merror:0.304\teval-merror:0.3297\ttrain-acc:0.696\teval-acc:0.6703\n",
      "[29]\ttrain-merror:0.303875\teval-merror:0.3296\ttrain-acc:0.696125\teval-acc:0.6704\n",
      "[30]\ttrain-merror:0.303863\teval-merror:0.32975\ttrain-acc:0.696137\teval-acc:0.67025\n",
      "[31]\ttrain-merror:0.30395\teval-merror:0.32955\ttrain-acc:0.69605\teval-acc:0.67045\n",
      "[32]\ttrain-merror:0.303937\teval-merror:0.3295\ttrain-acc:0.696063\teval-acc:0.6705\n",
      "[33]\ttrain-merror:0.303613\teval-merror:0.3298\ttrain-acc:0.696388\teval-acc:0.6702\n",
      "[34]\ttrain-merror:0.303762\teval-merror:0.3297\ttrain-acc:0.696237\teval-acc:0.6703\n",
      "[35]\ttrain-merror:0.303737\teval-merror:0.3295\ttrain-acc:0.696263\teval-acc:0.6705\n",
      "[36]\ttrain-merror:0.303625\teval-merror:0.32945\ttrain-acc:0.696375\teval-acc:0.67055\n",
      "[37]\ttrain-merror:0.303125\teval-merror:0.32905\ttrain-acc:0.696875\teval-acc:0.67095\n",
      "[38]\ttrain-merror:0.30305\teval-merror:0.3294\ttrain-acc:0.69695\teval-acc:0.6706\n",
      "[39]\ttrain-merror:0.302788\teval-merror:0.3294\ttrain-acc:0.697213\teval-acc:0.6706\n",
      "[40]\ttrain-merror:0.302713\teval-merror:0.3291\ttrain-acc:0.697287\teval-acc:0.6709\n",
      "[41]\ttrain-merror:0.302613\teval-merror:0.3292\ttrain-acc:0.697388\teval-acc:0.6708\n",
      "[42]\ttrain-merror:0.302475\teval-merror:0.32895\ttrain-acc:0.697525\teval-acc:0.67105\n",
      "[43]\ttrain-merror:0.302475\teval-merror:0.32955\ttrain-acc:0.697525\teval-acc:0.67045\n",
      "[44]\ttrain-merror:0.302363\teval-merror:0.32905\ttrain-acc:0.697638\teval-acc:0.67095\n",
      "[45]\ttrain-merror:0.302312\teval-merror:0.32915\ttrain-acc:0.697688\teval-acc:0.67085\n",
      "[46]\ttrain-merror:0.302487\teval-merror:0.32895\ttrain-acc:0.697512\teval-acc:0.67105\n",
      "[47]\ttrain-merror:0.302213\teval-merror:0.32885\ttrain-acc:0.697788\teval-acc:0.67115\n",
      "[48]\ttrain-merror:0.302213\teval-merror:0.32945\ttrain-acc:0.697788\teval-acc:0.67055\n",
      "[49]\ttrain-merror:0.302112\teval-merror:0.32925\ttrain-acc:0.697887\teval-acc:0.67075\n",
      "[50]\ttrain-merror:0.302112\teval-merror:0.32905\ttrain-acc:0.697887\teval-acc:0.67095\n",
      "[51]\ttrain-merror:0.301937\teval-merror:0.3288\ttrain-acc:0.698063\teval-acc:0.6712\n",
      "[52]\ttrain-merror:0.30185\teval-merror:0.3285\ttrain-acc:0.69815\teval-acc:0.6715\n",
      "[53]\ttrain-merror:0.301863\teval-merror:0.32815\ttrain-acc:0.698137\teval-acc:0.67185\n",
      "[54]\ttrain-merror:0.30165\teval-merror:0.3282\ttrain-acc:0.69835\teval-acc:0.6718\n",
      "[55]\ttrain-merror:0.3016\teval-merror:0.3284\ttrain-acc:0.6984\teval-acc:0.6716\n",
      "[56]\ttrain-merror:0.301525\teval-merror:0.32845\ttrain-acc:0.698475\teval-acc:0.67155\n",
      "[57]\ttrain-merror:0.301562\teval-merror:0.3285\ttrain-acc:0.698438\teval-acc:0.6715\n",
      "[58]\ttrain-merror:0.301437\teval-merror:0.328\ttrain-acc:0.698562\teval-acc:0.672\n",
      "[59]\ttrain-merror:0.301325\teval-merror:0.32805\ttrain-acc:0.698675\teval-acc:0.67195\n",
      "[60]\ttrain-merror:0.301237\teval-merror:0.32825\ttrain-acc:0.698762\teval-acc:0.67175\n",
      "[61]\ttrain-merror:0.3014\teval-merror:0.32775\ttrain-acc:0.6986\teval-acc:0.67225\n",
      "[62]\ttrain-merror:0.301062\teval-merror:0.3279\ttrain-acc:0.698937\teval-acc:0.6721\n",
      "[63]\ttrain-merror:0.300988\teval-merror:0.32795\ttrain-acc:0.699013\teval-acc:0.67205\n",
      "[64]\ttrain-merror:0.300887\teval-merror:0.32785\ttrain-acc:0.699113\teval-acc:0.67215\n",
      "[65]\ttrain-merror:0.30095\teval-merror:0.3282\ttrain-acc:0.69905\teval-acc:0.6718\n",
      "[66]\ttrain-merror:0.300875\teval-merror:0.3283\ttrain-acc:0.699125\teval-acc:0.6717\n",
      "[67]\ttrain-merror:0.300775\teval-merror:0.32835\ttrain-acc:0.699225\teval-acc:0.67165\n",
      "[68]\ttrain-merror:0.300687\teval-merror:0.3286\ttrain-acc:0.699313\teval-acc:0.6714\n",
      "[69]\ttrain-merror:0.30065\teval-merror:0.3283\ttrain-acc:0.69935\teval-acc:0.6717\n",
      "[70]\ttrain-merror:0.300463\teval-merror:0.32835\ttrain-acc:0.699538\teval-acc:0.67165\n",
      "[71]\ttrain-merror:0.300463\teval-merror:0.32835\ttrain-acc:0.699538\teval-acc:0.67165\n",
      "[72]\ttrain-merror:0.300512\teval-merror:0.32835\ttrain-acc:0.699488\teval-acc:0.67165\n",
      "[73]\ttrain-merror:0.300388\teval-merror:0.32805\ttrain-acc:0.699612\teval-acc:0.67195\n",
      "[74]\ttrain-merror:0.300213\teval-merror:0.3282\ttrain-acc:0.699788\teval-acc:0.6718\n",
      "[75]\ttrain-merror:0.3002\teval-merror:0.3279\ttrain-acc:0.6998\teval-acc:0.6721\n",
      "[76]\ttrain-merror:0.300175\teval-merror:0.3281\ttrain-acc:0.699825\teval-acc:0.6719\n",
      "[77]\ttrain-merror:0.300275\teval-merror:0.3281\ttrain-acc:0.699725\teval-acc:0.6719\n",
      "[78]\ttrain-merror:0.300088\teval-merror:0.3281\ttrain-acc:0.699913\teval-acc:0.6719\n",
      "[79]\ttrain-merror:0.299962\teval-merror:0.32825\ttrain-acc:0.700037\teval-acc:0.67175\n",
      "[80]\ttrain-merror:0.299875\teval-merror:0.32845\ttrain-acc:0.700125\teval-acc:0.67155\n",
      "[81]\ttrain-merror:0.299925\teval-merror:0.3286\ttrain-acc:0.700075\teval-acc:0.6714\n",
      "[82]\ttrain-merror:0.29975\teval-merror:0.32865\ttrain-acc:0.70025\teval-acc:0.67135\n",
      "[83]\ttrain-merror:0.2997\teval-merror:0.32835\ttrain-acc:0.7003\teval-acc:0.67165\n",
      "[84]\ttrain-merror:0.29955\teval-merror:0.3286\ttrain-acc:0.70045\teval-acc:0.6714\n",
      "[85]\ttrain-merror:0.2995\teval-merror:0.32845\ttrain-acc:0.7005\teval-acc:0.67155\n",
      "[86]\ttrain-merror:0.299513\teval-merror:0.3287\ttrain-acc:0.700488\teval-acc:0.6713\n",
      "Stopping. Best iteration:\n",
      "[61]\ttrain-merror:0.3014\teval-merror:0.32775\ttrain-acc:0.6986\teval-acc:0.67225\n",
      "\n",
      "age\n",
      "[0]\ttrain-merror:0.367813\teval-merror:0.37775\ttrain-acc:0.632188\teval-acc:0.62225\n",
      "Multiple eval metrics have been passed: 'eval-acc' will be used for early stopping.\n",
      "\n",
      "Will train until eval-acc hasn't improved in 25 rounds.\n",
      "[1]\ttrain-merror:0.361663\teval-merror:0.3715\ttrain-acc:0.638338\teval-acc:0.6285\n",
      "[2]\ttrain-merror:0.3599\teval-merror:0.36875\ttrain-acc:0.6401\teval-acc:0.63125\n",
      "[3]\ttrain-merror:0.35855\teval-merror:0.3675\ttrain-acc:0.64145\teval-acc:0.6325\n",
      "[4]\ttrain-merror:0.357525\teval-merror:0.36685\ttrain-acc:0.642475\teval-acc:0.63315\n",
      "[5]\ttrain-merror:0.356887\teval-merror:0.3664\ttrain-acc:0.643112\teval-acc:0.6336\n",
      "[6]\ttrain-merror:0.357038\teval-merror:0.36775\ttrain-acc:0.642962\teval-acc:0.63225\n",
      "[7]\ttrain-merror:0.356575\teval-merror:0.3662\ttrain-acc:0.643425\teval-acc:0.6338\n",
      "[8]\ttrain-merror:0.356225\teval-merror:0.3657\ttrain-acc:0.643775\teval-acc:0.6343\n",
      "[9]\ttrain-merror:0.356075\teval-merror:0.3661\ttrain-acc:0.643925\teval-acc:0.6339\n",
      "[10]\ttrain-merror:0.3558\teval-merror:0.36585\ttrain-acc:0.6442\teval-acc:0.63415\n",
      "[11]\ttrain-merror:0.355038\teval-merror:0.36595\ttrain-acc:0.644962\teval-acc:0.63405\n",
      "[12]\ttrain-merror:0.35485\teval-merror:0.3657\ttrain-acc:0.64515\teval-acc:0.6343\n",
      "[13]\ttrain-merror:0.3554\teval-merror:0.36545\ttrain-acc:0.6446\teval-acc:0.63455\n",
      "[14]\ttrain-merror:0.354775\teval-merror:0.3647\ttrain-acc:0.645225\teval-acc:0.6353\n",
      "[15]\ttrain-merror:0.355087\teval-merror:0.365\ttrain-acc:0.644912\teval-acc:0.635\n",
      "[16]\ttrain-merror:0.35535\teval-merror:0.36535\ttrain-acc:0.64465\teval-acc:0.63465\n",
      "[17]\ttrain-merror:0.354988\teval-merror:0.3651\ttrain-acc:0.645012\teval-acc:0.6349\n",
      "[18]\ttrain-merror:0.354875\teval-merror:0.3652\ttrain-acc:0.645125\teval-acc:0.6348\n",
      "[19]\ttrain-merror:0.354925\teval-merror:0.3652\ttrain-acc:0.645075\teval-acc:0.6348\n",
      "[20]\ttrain-merror:0.354712\teval-merror:0.36545\ttrain-acc:0.645288\teval-acc:0.63455\n",
      "[21]\ttrain-merror:0.3547\teval-merror:0.3656\ttrain-acc:0.6453\teval-acc:0.6344\n",
      "[22]\ttrain-merror:0.354475\teval-merror:0.36535\ttrain-acc:0.645525\teval-acc:0.63465\n",
      "[23]\ttrain-merror:0.3541\teval-merror:0.3656\ttrain-acc:0.6459\teval-acc:0.6344\n",
      "[24]\ttrain-merror:0.3541\teval-merror:0.3649\ttrain-acc:0.6459\teval-acc:0.6351\n",
      "[25]\ttrain-merror:0.354138\teval-merror:0.366\ttrain-acc:0.645863\teval-acc:0.634\n",
      "[26]\ttrain-merror:0.35385\teval-merror:0.36555\ttrain-acc:0.64615\teval-acc:0.63445\n",
      "[27]\ttrain-merror:0.35345\teval-merror:0.3646\ttrain-acc:0.64655\teval-acc:0.6354\n",
      "[28]\ttrain-merror:0.353388\teval-merror:0.36475\ttrain-acc:0.646613\teval-acc:0.63525\n",
      "[29]\ttrain-merror:0.353375\teval-merror:0.3647\ttrain-acc:0.646625\teval-acc:0.6353\n",
      "[30]\ttrain-merror:0.353588\teval-merror:0.3643\ttrain-acc:0.646412\teval-acc:0.6357\n",
      "[31]\ttrain-merror:0.3532\teval-merror:0.3647\ttrain-acc:0.6468\teval-acc:0.6353\n",
      "[32]\ttrain-merror:0.353225\teval-merror:0.36475\ttrain-acc:0.646775\teval-acc:0.63525\n",
      "[33]\ttrain-merror:0.3531\teval-merror:0.3641\ttrain-acc:0.6469\teval-acc:0.6359\n",
      "[34]\ttrain-merror:0.353013\teval-merror:0.36385\ttrain-acc:0.646988\teval-acc:0.63615\n",
      "[35]\ttrain-merror:0.35295\teval-merror:0.36425\ttrain-acc:0.64705\teval-acc:0.63575\n",
      "[36]\ttrain-merror:0.35285\teval-merror:0.36425\ttrain-acc:0.64715\teval-acc:0.63575\n",
      "[37]\ttrain-merror:0.352762\teval-merror:0.3642\ttrain-acc:0.647238\teval-acc:0.6358\n",
      "[38]\ttrain-merror:0.352562\teval-merror:0.3644\ttrain-acc:0.647437\teval-acc:0.6356\n",
      "[39]\ttrain-merror:0.352525\teval-merror:0.3641\ttrain-acc:0.647475\teval-acc:0.6359\n",
      "[40]\ttrain-merror:0.3525\teval-merror:0.3646\ttrain-acc:0.6475\teval-acc:0.6354\n",
      "[41]\ttrain-merror:0.352612\teval-merror:0.364\ttrain-acc:0.647388\teval-acc:0.636\n",
      "[42]\ttrain-merror:0.352325\teval-merror:0.36405\ttrain-acc:0.647675\teval-acc:0.63595\n",
      "[43]\ttrain-merror:0.352363\teval-merror:0.36415\ttrain-acc:0.647637\teval-acc:0.63585\n",
      "[44]\ttrain-merror:0.352313\teval-merror:0.36415\ttrain-acc:0.647687\teval-acc:0.63585\n",
      "[45]\ttrain-merror:0.352262\teval-merror:0.3638\ttrain-acc:0.647737\teval-acc:0.6362\n",
      "[46]\ttrain-merror:0.35215\teval-merror:0.3641\ttrain-acc:0.64785\teval-acc:0.6359\n",
      "[47]\ttrain-merror:0.351862\teval-merror:0.36415\ttrain-acc:0.648138\teval-acc:0.63585\n",
      "[48]\ttrain-merror:0.351788\teval-merror:0.3643\ttrain-acc:0.648212\teval-acc:0.6357\n",
      "[49]\ttrain-merror:0.351775\teval-merror:0.3642\ttrain-acc:0.648225\teval-acc:0.6358\n",
      "[50]\ttrain-merror:0.351737\teval-merror:0.364\ttrain-acc:0.648262\teval-acc:0.636\n",
      "[51]\ttrain-merror:0.351475\teval-merror:0.36415\ttrain-acc:0.648525\teval-acc:0.63585\n",
      "[52]\ttrain-merror:0.3515\teval-merror:0.36395\ttrain-acc:0.6485\teval-acc:0.63605\n",
      "[53]\ttrain-merror:0.351712\teval-merror:0.364\ttrain-acc:0.648288\teval-acc:0.636\n",
      "[54]\ttrain-merror:0.351625\teval-merror:0.36385\ttrain-acc:0.648375\teval-acc:0.63615\n",
      "[55]\ttrain-merror:0.35135\teval-merror:0.3639\ttrain-acc:0.64865\teval-acc:0.6361\n",
      "[56]\ttrain-merror:0.351388\teval-merror:0.36395\ttrain-acc:0.648613\teval-acc:0.63605\n",
      "[57]\ttrain-merror:0.351175\teval-merror:0.36425\ttrain-acc:0.648825\teval-acc:0.63575\n",
      "[58]\ttrain-merror:0.351075\teval-merror:0.36405\ttrain-acc:0.648925\teval-acc:0.63595\n",
      "[59]\ttrain-merror:0.351213\teval-merror:0.364\ttrain-acc:0.648787\teval-acc:0.636\n",
      "[60]\ttrain-merror:0.351312\teval-merror:0.36415\ttrain-acc:0.648687\teval-acc:0.63585\n",
      "[61]\ttrain-merror:0.351425\teval-merror:0.3638\ttrain-acc:0.648575\teval-acc:0.6362\n",
      "[62]\ttrain-merror:0.351288\teval-merror:0.3639\ttrain-acc:0.648713\teval-acc:0.6361\n",
      "[63]\ttrain-merror:0.351213\teval-merror:0.36365\ttrain-acc:0.648787\teval-acc:0.63635\n",
      "[64]\ttrain-merror:0.351\teval-merror:0.3639\ttrain-acc:0.649\teval-acc:0.6361\n",
      "[65]\ttrain-merror:0.350962\teval-merror:0.3639\ttrain-acc:0.649038\teval-acc:0.6361\n",
      "[66]\ttrain-merror:0.35095\teval-merror:0.36345\ttrain-acc:0.64905\teval-acc:0.63655\n",
      "[67]\ttrain-merror:0.350775\teval-merror:0.36345\ttrain-acc:0.649225\teval-acc:0.63655\n",
      "[68]\ttrain-merror:0.350775\teval-merror:0.36355\ttrain-acc:0.649225\teval-acc:0.63645\n",
      "[69]\ttrain-merror:0.350762\teval-merror:0.3635\ttrain-acc:0.649238\teval-acc:0.6365\n",
      "[70]\ttrain-merror:0.350662\teval-merror:0.36345\ttrain-acc:0.649338\teval-acc:0.63655\n",
      "[71]\ttrain-merror:0.350425\teval-merror:0.3635\ttrain-acc:0.649575\teval-acc:0.6365\n",
      "[72]\ttrain-merror:0.3505\teval-merror:0.3632\ttrain-acc:0.6495\teval-acc:0.6368\n",
      "[73]\ttrain-merror:0.350462\teval-merror:0.3635\ttrain-acc:0.649537\teval-acc:0.6365\n",
      "[74]\ttrain-merror:0.350375\teval-merror:0.36345\ttrain-acc:0.649625\teval-acc:0.63655\n",
      "[75]\ttrain-merror:0.350363\teval-merror:0.3637\ttrain-acc:0.649637\teval-acc:0.6363\n",
      "[76]\ttrain-merror:0.350237\teval-merror:0.3639\ttrain-acc:0.649763\teval-acc:0.6361\n",
      "[77]\ttrain-merror:0.350338\teval-merror:0.3637\ttrain-acc:0.649663\teval-acc:0.6363\n",
      "[78]\ttrain-merror:0.35025\teval-merror:0.3637\ttrain-acc:0.64975\teval-acc:0.6363\n",
      "[79]\ttrain-merror:0.3502\teval-merror:0.3638\ttrain-acc:0.6498\teval-acc:0.6362\n",
      "[80]\ttrain-merror:0.34985\teval-merror:0.3636\ttrain-acc:0.65015\teval-acc:0.6364\n",
      "[81]\ttrain-merror:0.349937\teval-merror:0.3637\ttrain-acc:0.650062\teval-acc:0.6363\n",
      "[82]\ttrain-merror:0.35\teval-merror:0.36355\ttrain-acc:0.65\teval-acc:0.63645\n",
      "[83]\ttrain-merror:0.350087\teval-merror:0.3636\ttrain-acc:0.649913\teval-acc:0.6364\n",
      "[84]\ttrain-merror:0.34995\teval-merror:0.3634\ttrain-acc:0.65005\teval-acc:0.6366\n",
      "[85]\ttrain-merror:0.350062\teval-merror:0.3635\ttrain-acc:0.649937\teval-acc:0.6365\n",
      "[86]\ttrain-merror:0.349988\teval-merror:0.36335\ttrain-acc:0.650012\teval-acc:0.63665\n",
      "[87]\ttrain-merror:0.349862\teval-merror:0.36315\ttrain-acc:0.650138\teval-acc:0.63685\n",
      "[88]\ttrain-merror:0.3498\teval-merror:0.36325\ttrain-acc:0.6502\teval-acc:0.63675\n",
      "[89]\ttrain-merror:0.349712\teval-merror:0.3632\ttrain-acc:0.650288\teval-acc:0.6368\n",
      "[90]\ttrain-merror:0.3497\teval-merror:0.3631\ttrain-acc:0.6503\teval-acc:0.6369\n",
      "[91]\ttrain-merror:0.349512\teval-merror:0.363\ttrain-acc:0.650487\teval-acc:0.637\n",
      "[92]\ttrain-merror:0.349425\teval-merror:0.36325\ttrain-acc:0.650575\teval-acc:0.63675\n",
      "[93]\ttrain-merror:0.3494\teval-merror:0.36295\ttrain-acc:0.6506\teval-acc:0.63705\n",
      "[94]\ttrain-merror:0.349413\teval-merror:0.36315\ttrain-acc:0.650587\teval-acc:0.63685\n",
      "[95]\ttrain-merror:0.349488\teval-merror:0.3631\ttrain-acc:0.650513\teval-acc:0.6369\n",
      "[96]\ttrain-merror:0.349488\teval-merror:0.36305\ttrain-acc:0.650513\teval-acc:0.63695\n",
      "[97]\ttrain-merror:0.349512\teval-merror:0.3629\ttrain-acc:0.650487\teval-acc:0.6371\n",
      "[98]\ttrain-merror:0.349537\teval-merror:0.36285\ttrain-acc:0.650463\teval-acc:0.63715\n",
      "[99]\ttrain-merror:0.349488\teval-merror:0.363\ttrain-acc:0.650513\teval-acc:0.637\n",
      "[100]\ttrain-merror:0.3494\teval-merror:0.3631\ttrain-acc:0.6506\teval-acc:0.6369\n",
      "[101]\ttrain-merror:0.349212\teval-merror:0.3629\ttrain-acc:0.650787\teval-acc:0.6371\n",
      "[102]\ttrain-merror:0.349138\teval-merror:0.36335\ttrain-acc:0.650863\teval-acc:0.63665\n",
      "[103]\ttrain-merror:0.348975\teval-merror:0.3636\ttrain-acc:0.651025\teval-acc:0.6364\n",
      "[104]\ttrain-merror:0.348987\teval-merror:0.3635\ttrain-acc:0.651012\teval-acc:0.6365\n",
      "[105]\ttrain-merror:0.348925\teval-merror:0.3634\ttrain-acc:0.651075\teval-acc:0.6366\n",
      "[106]\ttrain-merror:0.348925\teval-merror:0.3635\ttrain-acc:0.651075\teval-acc:0.6365\n",
      "[107]\ttrain-merror:0.3488\teval-merror:0.3636\ttrain-acc:0.6512\teval-acc:0.6364\n",
      "[108]\ttrain-merror:0.348637\teval-merror:0.3634\ttrain-acc:0.651362\teval-acc:0.6366\n",
      "[109]\ttrain-merror:0.348637\teval-merror:0.3635\ttrain-acc:0.651362\teval-acc:0.6365\n",
      "[110]\ttrain-merror:0.348588\teval-merror:0.3634\ttrain-acc:0.651412\teval-acc:0.6366\n",
      "[111]\ttrain-merror:0.348538\teval-merror:0.36365\ttrain-acc:0.651463\teval-acc:0.63635\n",
      "[112]\ttrain-merror:0.348525\teval-merror:0.3631\ttrain-acc:0.651475\teval-acc:0.6369\n",
      "[113]\ttrain-merror:0.348525\teval-merror:0.36295\ttrain-acc:0.651475\teval-acc:0.63705\n",
      "[114]\ttrain-merror:0.348575\teval-merror:0.36275\ttrain-acc:0.651425\teval-acc:0.63725\n",
      "[115]\ttrain-merror:0.348525\teval-merror:0.36275\ttrain-acc:0.651475\teval-acc:0.63725\n",
      "[116]\ttrain-merror:0.3486\teval-merror:0.36295\ttrain-acc:0.6514\teval-acc:0.63705\n",
      "[117]\ttrain-merror:0.348475\teval-merror:0.36265\ttrain-acc:0.651525\teval-acc:0.63735\n",
      "[118]\ttrain-merror:0.348425\teval-merror:0.36265\ttrain-acc:0.651575\teval-acc:0.63735\n",
      "[119]\ttrain-merror:0.3484\teval-merror:0.3627\ttrain-acc:0.6516\teval-acc:0.6373\n",
      "[120]\ttrain-merror:0.348487\teval-merror:0.36245\ttrain-acc:0.651513\teval-acc:0.63755\n",
      "[121]\ttrain-merror:0.348437\teval-merror:0.36275\ttrain-acc:0.651563\teval-acc:0.63725\n",
      "[122]\ttrain-merror:0.3483\teval-merror:0.36285\ttrain-acc:0.6517\teval-acc:0.63715\n",
      "[123]\ttrain-merror:0.34825\teval-merror:0.3629\ttrain-acc:0.65175\teval-acc:0.6371\n",
      "[124]\ttrain-merror:0.348112\teval-merror:0.3627\ttrain-acc:0.651887\teval-acc:0.6373\n",
      "[125]\ttrain-merror:0.348238\teval-merror:0.3631\ttrain-acc:0.651763\teval-acc:0.6369\n",
      "[126]\ttrain-merror:0.348238\teval-merror:0.36305\ttrain-acc:0.651763\teval-acc:0.63695\n",
      "[127]\ttrain-merror:0.348\teval-merror:0.36325\ttrain-acc:0.652\teval-acc:0.63675\n",
      "[128]\ttrain-merror:0.348038\teval-merror:0.3633\ttrain-acc:0.651963\teval-acc:0.6367\n",
      "[129]\ttrain-merror:0.3479\teval-merror:0.3632\ttrain-acc:0.6521\teval-acc:0.6368\n",
      "[130]\ttrain-merror:0.347813\teval-merror:0.363\ttrain-acc:0.652188\teval-acc:0.637\n",
      "[131]\ttrain-merror:0.347712\teval-merror:0.3631\ttrain-acc:0.652288\teval-acc:0.6369\n",
      "[132]\ttrain-merror:0.347688\teval-merror:0.363\ttrain-acc:0.652312\teval-acc:0.637\n",
      "[133]\ttrain-merror:0.347675\teval-merror:0.3629\ttrain-acc:0.652325\teval-acc:0.6371\n",
      "[134]\ttrain-merror:0.347575\teval-merror:0.36285\ttrain-acc:0.652425\teval-acc:0.63715\n",
      "[135]\ttrain-merror:0.347488\teval-merror:0.363\ttrain-acc:0.652513\teval-acc:0.637\n",
      "[136]\ttrain-merror:0.34745\teval-merror:0.3628\ttrain-acc:0.65255\teval-acc:0.6372\n",
      "[137]\ttrain-merror:0.347387\teval-merror:0.3631\ttrain-acc:0.652613\teval-acc:0.6369\n",
      "[138]\ttrain-merror:0.347313\teval-merror:0.36325\ttrain-acc:0.652687\teval-acc:0.63675\n",
      "[139]\ttrain-merror:0.347275\teval-merror:0.36325\ttrain-acc:0.652725\teval-acc:0.63675\n",
      "[140]\ttrain-merror:0.347237\teval-merror:0.3631\ttrain-acc:0.652763\teval-acc:0.6369\n",
      "[141]\ttrain-merror:0.34735\teval-merror:0.36325\ttrain-acc:0.65265\teval-acc:0.63675\n",
      "[142]\ttrain-merror:0.347362\teval-merror:0.3633\ttrain-acc:0.652637\teval-acc:0.6367\n",
      "[143]\ttrain-merror:0.347362\teval-merror:0.36355\ttrain-acc:0.652637\teval-acc:0.63645\n",
      "[144]\ttrain-merror:0.347325\teval-merror:0.36315\ttrain-acc:0.652675\teval-acc:0.63685\n",
      "[145]\ttrain-merror:0.34715\teval-merror:0.3631\ttrain-acc:0.65285\teval-acc:0.6369\n",
      "Stopping. Best iteration:\n",
      "[120]\ttrain-merror:0.348487\teval-merror:0.36245\ttrain-acc:0.651513\teval-acc:0.63755\n",
      "\n",
      "gender\n",
      "[0]\ttrain-merror:0.148525\teval-merror:0.1558\ttrain-acc:0.851475\teval-acc:0.8442\n",
      "Multiple eval metrics have been passed: 'eval-acc' will be used for early stopping.\n",
      "\n",
      "Will train until eval-acc hasn't improved in 25 rounds.\n",
      "[1]\ttrain-merror:0.147425\teval-merror:0.15315\ttrain-acc:0.852575\teval-acc:0.84685\n",
      "[2]\ttrain-merror:0.146525\teval-merror:0.1527\ttrain-acc:0.853475\teval-acc:0.8473\n",
      "[3]\ttrain-merror:0.1463\teval-merror:0.15185\ttrain-acc:0.8537\teval-acc:0.84815\n",
      "[4]\ttrain-merror:0.146288\teval-merror:0.15185\ttrain-acc:0.853712\teval-acc:0.84815\n",
      "[5]\ttrain-merror:0.14595\teval-merror:0.1511\ttrain-acc:0.85405\teval-acc:0.8489\n",
      "[6]\ttrain-merror:0.14605\teval-merror:0.15085\ttrain-acc:0.85395\teval-acc:0.84915\n",
      "[7]\ttrain-merror:0.145887\teval-merror:0.15105\ttrain-acc:0.854113\teval-acc:0.84895\n",
      "[8]\ttrain-merror:0.14575\teval-merror:0.151\ttrain-acc:0.85425\teval-acc:0.849\n",
      "[9]\ttrain-merror:0.145537\teval-merror:0.15085\ttrain-acc:0.854463\teval-acc:0.84915\n",
      "[10]\ttrain-merror:0.145325\teval-merror:0.15065\ttrain-acc:0.854675\teval-acc:0.84935\n",
      "[11]\ttrain-merror:0.145475\teval-merror:0.15095\ttrain-acc:0.854525\teval-acc:0.84905\n",
      "[12]\ttrain-merror:0.145287\teval-merror:0.15115\ttrain-acc:0.854712\teval-acc:0.84885\n",
      "[13]\ttrain-merror:0.145187\teval-merror:0.1512\ttrain-acc:0.854812\teval-acc:0.8488\n",
      "[14]\ttrain-merror:0.14505\teval-merror:0.15125\ttrain-acc:0.85495\teval-acc:0.84875\n",
      "[15]\ttrain-merror:0.14515\teval-merror:0.1511\ttrain-acc:0.85485\teval-acc:0.8489\n",
      "[16]\ttrain-merror:0.144938\teval-merror:0.15135\ttrain-acc:0.855062\teval-acc:0.84865\n",
      "[17]\ttrain-merror:0.144938\teval-merror:0.15135\ttrain-acc:0.855062\teval-acc:0.84865\n",
      "[18]\ttrain-merror:0.145025\teval-merror:0.15155\ttrain-acc:0.854975\teval-acc:0.84845\n",
      "[19]\ttrain-merror:0.145138\teval-merror:0.15115\ttrain-acc:0.854862\teval-acc:0.84885\n",
      "[20]\ttrain-merror:0.145038\teval-merror:0.1511\ttrain-acc:0.854962\teval-acc:0.8489\n",
      "[21]\ttrain-merror:0.145112\teval-merror:0.15095\ttrain-acc:0.854888\teval-acc:0.84905\n",
      "[22]\ttrain-merror:0.145063\teval-merror:0.15085\ttrain-acc:0.854938\teval-acc:0.84915\n",
      "[23]\ttrain-merror:0.145087\teval-merror:0.151\ttrain-acc:0.854912\teval-acc:0.849\n",
      "[24]\ttrain-merror:0.145087\teval-merror:0.1507\ttrain-acc:0.854912\teval-acc:0.8493\n",
      "[25]\ttrain-merror:0.144875\teval-merror:0.1507\ttrain-acc:0.855125\teval-acc:0.8493\n",
      "[26]\ttrain-merror:0.144837\teval-merror:0.1507\ttrain-acc:0.855163\teval-acc:0.8493\n",
      "[27]\ttrain-merror:0.144938\teval-merror:0.15065\ttrain-acc:0.855062\teval-acc:0.84935\n",
      "[28]\ttrain-merror:0.1449\teval-merror:0.15075\ttrain-acc:0.8551\teval-acc:0.84925\n",
      "[29]\ttrain-merror:0.145\teval-merror:0.1506\ttrain-acc:0.855\teval-acc:0.8494\n",
      "[30]\ttrain-merror:0.144812\teval-merror:0.15065\ttrain-acc:0.855187\teval-acc:0.84935\n",
      "[31]\ttrain-merror:0.144725\teval-merror:0.1507\ttrain-acc:0.855275\teval-acc:0.8493\n",
      "[32]\ttrain-merror:0.14485\teval-merror:0.1506\ttrain-acc:0.85515\teval-acc:0.8494\n",
      "[33]\ttrain-merror:0.144788\teval-merror:0.1505\ttrain-acc:0.855213\teval-acc:0.8495\n",
      "[34]\ttrain-merror:0.144637\teval-merror:0.15045\ttrain-acc:0.855363\teval-acc:0.84955\n",
      "[35]\ttrain-merror:0.144812\teval-merror:0.15045\ttrain-acc:0.855187\teval-acc:0.84955\n",
      "[36]\ttrain-merror:0.144925\teval-merror:0.15075\ttrain-acc:0.855075\teval-acc:0.84925\n",
      "[37]\ttrain-merror:0.144987\teval-merror:0.1508\ttrain-acc:0.855012\teval-acc:0.8492\n",
      "[38]\ttrain-merror:0.145025\teval-merror:0.1508\ttrain-acc:0.854975\teval-acc:0.8492\n",
      "[39]\ttrain-merror:0.144863\teval-merror:0.15065\ttrain-acc:0.855137\teval-acc:0.84935\n",
      "[40]\ttrain-merror:0.144875\teval-merror:0.1505\ttrain-acc:0.855125\teval-acc:0.8495\n",
      "[41]\ttrain-merror:0.144863\teval-merror:0.1505\ttrain-acc:0.855137\teval-acc:0.8495\n",
      "[42]\ttrain-merror:0.1448\teval-merror:0.1504\ttrain-acc:0.8552\teval-acc:0.8496\n",
      "[43]\ttrain-merror:0.144675\teval-merror:0.1506\ttrain-acc:0.855325\teval-acc:0.8494\n",
      "[44]\ttrain-merror:0.144662\teval-merror:0.15065\ttrain-acc:0.855337\teval-acc:0.84935\n",
      "[45]\ttrain-merror:0.144575\teval-merror:0.15055\ttrain-acc:0.855425\teval-acc:0.84945\n",
      "[46]\ttrain-merror:0.144588\teval-merror:0.15035\ttrain-acc:0.855413\teval-acc:0.84965\n",
      "[47]\ttrain-merror:0.144625\teval-merror:0.15045\ttrain-acc:0.855375\teval-acc:0.84955\n",
      "[48]\ttrain-merror:0.14455\teval-merror:0.1504\ttrain-acc:0.85545\teval-acc:0.8496\n",
      "[49]\ttrain-merror:0.144488\teval-merror:0.1502\ttrain-acc:0.855513\teval-acc:0.8498\n",
      "[50]\ttrain-merror:0.144513\teval-merror:0.1502\ttrain-acc:0.855487\teval-acc:0.8498\n",
      "[51]\ttrain-merror:0.14455\teval-merror:0.15035\ttrain-acc:0.85545\teval-acc:0.84965\n",
      "[52]\ttrain-merror:0.144425\teval-merror:0.15025\ttrain-acc:0.855575\teval-acc:0.84975\n",
      "[53]\ttrain-merror:0.144488\teval-merror:0.1503\ttrain-acc:0.855513\teval-acc:0.8497\n",
      "[54]\ttrain-merror:0.144425\teval-merror:0.1502\ttrain-acc:0.855575\teval-acc:0.8498\n",
      "[55]\ttrain-merror:0.14445\teval-merror:0.15035\ttrain-acc:0.85555\teval-acc:0.84965\n",
      "[56]\ttrain-merror:0.144438\teval-merror:0.1504\ttrain-acc:0.855563\teval-acc:0.8496\n",
      "[57]\ttrain-merror:0.144375\teval-merror:0.15025\ttrain-acc:0.855625\teval-acc:0.84975\n",
      "[58]\ttrain-merror:0.144362\teval-merror:0.15015\ttrain-acc:0.855638\teval-acc:0.84985\n",
      "[59]\ttrain-merror:0.14435\teval-merror:0.15035\ttrain-acc:0.85565\teval-acc:0.84965\n",
      "[60]\ttrain-merror:0.144362\teval-merror:0.1503\ttrain-acc:0.855638\teval-acc:0.8497\n",
      "[61]\ttrain-merror:0.144275\teval-merror:0.1505\ttrain-acc:0.855725\teval-acc:0.8495\n",
      "[62]\ttrain-merror:0.144212\teval-merror:0.15045\ttrain-acc:0.855788\teval-acc:0.84955\n",
      "[63]\ttrain-merror:0.144287\teval-merror:0.1502\ttrain-acc:0.855712\teval-acc:0.8498\n",
      "[64]\ttrain-merror:0.1443\teval-merror:0.1503\ttrain-acc:0.8557\teval-acc:0.8497\n",
      "[65]\ttrain-merror:0.144287\teval-merror:0.1503\ttrain-acc:0.855712\teval-acc:0.8497\n",
      "[66]\ttrain-merror:0.144212\teval-merror:0.15035\ttrain-acc:0.855788\teval-acc:0.84965\n",
      "[67]\ttrain-merror:0.144262\teval-merror:0.1501\ttrain-acc:0.855738\teval-acc:0.8499\n",
      "[68]\ttrain-merror:0.1442\teval-merror:0.1501\ttrain-acc:0.8558\teval-acc:0.8499\n",
      "[69]\ttrain-merror:0.144212\teval-merror:0.15005\ttrain-acc:0.855788\teval-acc:0.84995\n",
      "[70]\ttrain-merror:0.144163\teval-merror:0.1501\ttrain-acc:0.855838\teval-acc:0.8499\n",
      "[71]\ttrain-merror:0.144125\teval-merror:0.15025\ttrain-acc:0.855875\teval-acc:0.84975\n",
      "[72]\ttrain-merror:0.144075\teval-merror:0.15\ttrain-acc:0.855925\teval-acc:0.85\n",
      "[73]\ttrain-merror:0.144138\teval-merror:0.15\ttrain-acc:0.855862\teval-acc:0.85\n",
      "[74]\ttrain-merror:0.144112\teval-merror:0.15035\ttrain-acc:0.855888\teval-acc:0.84965\n",
      "[75]\ttrain-merror:0.144138\teval-merror:0.15045\ttrain-acc:0.855862\teval-acc:0.84955\n",
      "[76]\ttrain-merror:0.1442\teval-merror:0.15045\ttrain-acc:0.8558\teval-acc:0.84955\n",
      "[77]\ttrain-merror:0.144262\teval-merror:0.1504\ttrain-acc:0.855738\teval-acc:0.8496\n",
      "[78]\ttrain-merror:0.144163\teval-merror:0.1503\ttrain-acc:0.855838\teval-acc:0.8497\n",
      "[79]\ttrain-merror:0.14415\teval-merror:0.1504\ttrain-acc:0.85585\teval-acc:0.8496\n",
      "[80]\ttrain-merror:0.144138\teval-merror:0.1504\ttrain-acc:0.855862\teval-acc:0.8496\n",
      "[81]\ttrain-merror:0.1442\teval-merror:0.15025\ttrain-acc:0.8558\teval-acc:0.84975\n",
      "[82]\ttrain-merror:0.14405\teval-merror:0.15025\ttrain-acc:0.85595\teval-acc:0.84975\n",
      "[83]\ttrain-merror:0.144025\teval-merror:0.15035\ttrain-acc:0.855975\teval-acc:0.84965\n",
      "[84]\ttrain-merror:0.144025\teval-merror:0.15045\ttrain-acc:0.855975\teval-acc:0.84955\n",
      "[85]\ttrain-merror:0.144063\teval-merror:0.1504\ttrain-acc:0.855938\teval-acc:0.8496\n",
      "[86]\ttrain-merror:0.14395\teval-merror:0.15045\ttrain-acc:0.85605\teval-acc:0.84955\n",
      "[87]\ttrain-merror:0.143975\teval-merror:0.15035\ttrain-acc:0.856025\teval-acc:0.84965\n",
      "[88]\ttrain-merror:0.143937\teval-merror:0.15035\ttrain-acc:0.856062\teval-acc:0.84965\n",
      "[89]\ttrain-merror:0.143963\teval-merror:0.15035\ttrain-acc:0.856038\teval-acc:0.84965\n",
      "[90]\ttrain-merror:0.14395\teval-merror:0.1502\ttrain-acc:0.85605\teval-acc:0.8498\n",
      "[91]\ttrain-merror:0.143925\teval-merror:0.15025\ttrain-acc:0.856075\teval-acc:0.84975\n",
      "[92]\ttrain-merror:0.143875\teval-merror:0.1502\ttrain-acc:0.856125\teval-acc:0.8498\n",
      "[93]\ttrain-merror:0.143937\teval-merror:0.1503\ttrain-acc:0.856062\teval-acc:0.8497\n",
      "[94]\ttrain-merror:0.143888\teval-merror:0.15015\ttrain-acc:0.856113\teval-acc:0.84985\n",
      "[95]\ttrain-merror:0.143888\teval-merror:0.1503\ttrain-acc:0.856113\teval-acc:0.8497\n",
      "[96]\ttrain-merror:0.143813\teval-merror:0.15035\ttrain-acc:0.856187\teval-acc:0.84965\n",
      "[97]\ttrain-merror:0.143837\teval-merror:0.15035\ttrain-acc:0.856163\teval-acc:0.84965\n",
      "Stopping. Best iteration:\n",
      "[72]\ttrain-merror:0.144075\teval-merror:0.15\ttrain-acc:0.855925\teval-acc:0.85\n",
      "\n"
     ]
    }
   ],
   "source": [
    "'''xgb-ens for education/age/gender'''\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import xgboost as xgb\n",
    "import cfg\n",
    "import datetime\n",
    "\n",
    "def xgb_acc_score(preds,dtrain):\n",
    "    y_true = dtrain.get_label()\n",
    "    y_pred = np.argmax(preds,axis=1)\n",
    "    return [('acc',np.mean(y_true == y_pred))]\n",
    "\n",
    "df_lr = pd.read_csv(cfg.data_path + 'tfidf_stack_10W.csv')\n",
    "df_dm = pd.read_csv(cfg.data_path + 'dmd2v_stack_10W.csv')\n",
    "df_dbow = pd.read_csv(cfg.data_path + 'dbowd2v_stack_10W.csv')\n",
    "\n",
    "df_lb = pd.read_csv(cfg.data_path + 'all.csv',usecols=['Id','Education','age','gender'],nrows=100000)\n",
    "ys = {}\n",
    "for lb in ['Education','age','gender']:\n",
    "    ys[lb] = np.array(df_lb[lb])\n",
    "\n",
    "'''最好的参数组合'''\n",
    "#-------------------------education----------------------------------\n",
    "TR = 80000\n",
    "df_sub = pd.DataFrame()\n",
    "df_sub['Id'] = df_lb.iloc[TR:]['Id']\n",
    "seed = 10\n",
    "lb = 'Education'\n",
    "print(lb)\n",
    "\n",
    "esr = 25\n",
    "evals = 1\n",
    "n_trees = 1000\n",
    "\n",
    "df = pd.concat([df_lr,df_dbow,df_dm],axis=1)\n",
    "print(df.columns)\n",
    "num_class = len(pd.value_counts(ys[lb]))\n",
    "X = df.iloc[:TR]\n",
    "y = ys[lb][:TR]\n",
    "X_te = df.iloc[TR:]\n",
    "y_te = ys[lb][TR:]\n",
    "\n",
    "ss = 0.9\n",
    "mc = 2\n",
    "md = 8\n",
    "gm = 2\n",
    "# n_trees = 30\n",
    "\n",
    "params = {\n",
    "    \"objective\": \"multi:softprob\",\n",
    "    \"booster\": \"gbtree\",\n",
    "#     \"eval_metric\": \"merror\",\n",
    "    \"num_class\":num_class,\n",
    "    'max_depth':md,\n",
    "    'min_child_weight':mc,\n",
    "    'subsample':ss,\n",
    "    'colsample_bytree':0.8,\n",
    "    'gamma':gm,\n",
    "    \"eta\": 0.01,\n",
    "    \"lambda\":0,\n",
    "    'alpha':0,\n",
    "    \"silent\": 1,\n",
    "#     'seed':seed,\n",
    "}\n",
    "\n",
    "dtrain = xgb.DMatrix(X, y)\n",
    "dvalid = xgb.DMatrix(X_te, y_te)\n",
    "watchlist = [(dtrain, 'train'), (dvalid, 'eval')]\n",
    "bst = xgb.train(params, dtrain, n_trees, evals=watchlist,feval=xgb_acc_score,maximize=True,\n",
    "                early_stopping_rounds=esr, verbose_eval=evals)\n",
    "df_sub['Education'] = np.argmax(bst.predict(dvalid),axis=1) + 1\n",
    "#------------------------ age-----------------------------------\n",
    "lb = 'age'\n",
    "print(lb)\n",
    "num_class = len(pd.value_counts(ys[lb]))\n",
    "\n",
    "# df = pd.concat([df_stack_tfidf,df_stack_d2v],axis=1)\n",
    "num_class = len(pd.value_counts(ys[lb]))\n",
    "X = df.iloc[:TR]\n",
    "y = ys[lb][:TR]\n",
    "X_te = df.iloc[TR:]\n",
    "y_te = ys[lb][TR:]\n",
    "\n",
    "ss = 0.5\n",
    "mc = 3\n",
    "md = 7\n",
    "gm = 2\n",
    "# n_trees = 37\n",
    "\n",
    "params = {\n",
    "    \"objective\": \"multi:softprob\",\n",
    "    \"booster\": \"gbtree\",\n",
    "#     \"eval_metric\": \"merror\",\n",
    "    \"num_class\":num_class,\n",
    "    'max_depth':md,\n",
    "    'min_child_weight':mc,\n",
    "    'subsample':ss,\n",
    "    'colsample_bytree':1,\n",
    "    'gamma':gm,\n",
    "    \"eta\": 0.01,\n",
    "    \"lambda\":0,\n",
    "    'alpha':0,\n",
    "    \"silent\": 1,\n",
    "#     'seed':seed,\n",
    "}\n",
    "\n",
    "dtrain = xgb.DMatrix(X, y)\n",
    "dvalid = xgb.DMatrix(X_te, y_te)\n",
    "watchlist = [(dtrain, 'train'), (dvalid, 'eval')]\n",
    "bst = xgb.train(params, dtrain, n_trees, evals=watchlist,feval=xgb_acc_score,maximize=True,\n",
    "                early_stopping_rounds=esr, verbose_eval=evals)\n",
    "df_sub['age'] = np.argmax(bst.predict(dvalid),axis=1)+1\n",
    "#--------------------------gender-------------------------------------\n",
    "lb = 'gender'\n",
    "print(lb)\n",
    "num_class = len(pd.value_counts(ys[lb]))\n",
    "\n",
    "# df = pd.concat([df_lr,df_multid2v],axis=1)\n",
    "num_class = len(pd.value_counts(ys[lb]))\n",
    "X = df.iloc[:TR]\n",
    "y = ys[lb][:TR]\n",
    "X_te = df.iloc[TR:]\n",
    "y_te = ys[lb][TR:]\n",
    "\n",
    "\n",
    "ss = 0.5\n",
    "mc = 0.8\n",
    "md = 7\n",
    "gm = 1\n",
    "# n_trees = 25\n",
    "\n",
    "params = {\n",
    "    \"objective\": \"multi:softprob\",\n",
    "    \"booster\": \"gbtree\",\n",
    "#     \"eval_metric\": \"merror\",\n",
    "    \"num_class\":num_class,\n",
    "    'max_depth':md,\n",
    "    'min_child_weight':mc,\n",
    "    'subsample':ss,\n",
    "    'colsample_bytree':1,\n",
    "    'gamma':gm,\n",
    "    \"eta\": 0.01,\n",
    "    \"lambda\":0,\n",
    "    'alpha':0,\n",
    "    \"silent\": 1,\n",
    "#     'seed':seed,\n",
    "}\n",
    "\n",
    "dtrain = xgb.DMatrix(X, y)\n",
    "dvalid = xgb.DMatrix(X_te, y_te)\n",
    "watchlist = [(dtrain, 'train'), (dvalid, 'eval')]\n",
    "bst = xgb.train(params, dtrain, n_trees, evals=watchlist,feval=xgb_acc_score,maximize=True,\n",
    "                early_stopping_rounds=esr, verbose_eval=evals)\n",
    "df_sub['gender'] = np.argmax(bst.predict(dvalid),axis=1)+1\n",
    "\n",
    "df_sub = df_sub[['Id','age','gender','Education']]\n",
    "df_sub.to_csv(cfg.data_path + 'tfidf_dm_dbow_2W.csv',index=None,header=None,sep=' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
